Module redlite

Sub-modules

Functions

load_dataset

def load_dataset(
    name: str,
    split: Literal['test', 'train'] = 'test',
    transform: Optional[Callable] = None,
    **extra
) -> redlite._core.NamedDataset

Loads dataset. Downloads it to the local machine if necessary.

name (str): Dataset name. Starts with hub prefix "hf:" (HuggingFace datasets hub), or "inno:" (Innodata datasets hub)
split (str): Split name ("test" or "train")
transform (callable): A function that takes a dataset item and returns a transformed item (optional).
extra (dict[str,str]): Extra parameters passed to dataset loader.

Returns: NamedDataset object.

Sample usage:

dataset = load_dataset("hf:innodatalabs/rt-frank")

print(dataset.name)
print(dataset.split)

for record in dataset:
    ...

parallel_run

def parallel_run(
    *,
    model_producer: Callable[[], redlite._core.NamedModel],
    dataset: redlite._core.NamedDataset,
    metric_producer: Callable[[], redlite._core.NamedMetric],
    name: str | None = None,
    max_samples=0,
    num_workers: int = 64
) -> redlite._core.Run

rescore

def rescore(
    *,
    run: str,
    metric: redlite._core.NamedMetric,
    name: str | None = None,
    dry: bool = False
) -> redlite._core.Run

Uses a prior experiment and re-runs it with a different metric.

Model answers will not be re-computed, but each answer will be re-evaluated with the new metric. This is normally very fast.

run (str): The parent run.
metric (NamedMetric): Metric.
name (str, optional): The name of the run. It will automatically get a numeric suffix to ensure global uniqueness. If not provided, a unique name will be auto-generated.
dry (bool, optional): If set to True, does not write new run data to the disk. Only displays the aggregated metric on the screen. Useful for developing and debugging metrics.

Returns the experiment metadata as dict object. See Run docs for the structure.

Sample usage:

metric = MyNewMetric(...)

rescore(run="tired-tiger-32", metric=metric)

run

def run(
    *,
    model: redlite._core.NamedModel,
    dataset: redlite._core.NamedDataset,
    metric: redlite._core.NamedMetric,
    name: str | None = None,
    max_samples=0
) -> redlite._core.Run

Runs experiment, using the given model, dataset, and metric.

model (NamedModel): Model.
dataset (NamedDataset): Dataset.
metric (NamedMetric): Metric.
name (str, optional): The name of the run. It will automatically get a numeric suffix to ensure global uniqueness. If not provided, a unique name will be auto-generated.
max_samples (int, optional): Allows one to limit the number of samples in the run. Value of zero (the default) means "run the whole dataset".

Returns the run metadata as a dict object. See Run docs for the structure.

Sample usage:

model = MyModel(...)
dataset = MyDataset(...)
metric = MyMetric(...)

run(model=model, dataset=dataset, metric=metric)

Classes

DatasetItem

class DatasetItem(
    /,
    *args,
    **kwargs
)

Dataset item.

A plain dict with three required keys:

id (str): Uniqie id of this item. Uniqueness is across all splits of the dataset.
messages (list[Message]): Conversation messages. Last message is expected to have role="user".
expected (str): Expected response.

Ancestors (in MRO)

builtins.dict

Methods

clear

def clear(
    ...
)

D.clear() -> None. Remove all items from D.

copy

def copy(
    ...
)

D.copy() -> a shallow copy of D

fromkeys

def fromkeys(
    iterable,
    value=None,
    /
)

Create a new dictionary with keys from iterable and values set to value.

get

def get(
    self,
    key,
    default=None,
    /
)

Return the value for key if key is in the dictionary, else default.

items

def items(
    ...
)

D.items() -> a set-like object providing a view on D's items

keys

def keys(
    ...
)

D.keys() -> a set-like object providing a view on D's keys

pop

def pop(
    ...
)

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem

def popitem(
    self,
    /
)

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault

def setdefault(
    self,
    key,
    default=None,
    /
)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update

def update(
    ...
)

D.update([E, ]**F) -> None. Update D from dict/iterable E and F.

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values

def values(
    ...
)

D.values() -> an object providing a view on D's values

Message

class Message(
    /,
    *args,
    **kwargs
)

Message.

A plain dict with two keys:

role (str): one of "system", "user", or "assistant".
content (str): message content.

Ancestors (in MRO)

builtins.dict

Methods

clear

def clear(
    ...
)

D.clear() -> None. Remove all items from D.

copy

def copy(
    ...
)

D.copy() -> a shallow copy of D

fromkeys

def fromkeys(
    iterable,
    value=None,
    /
)

Create a new dictionary with keys from iterable and values set to value.

get

def get(
    self,
    key,
    default=None,
    /
)

Return the value for key if key is in the dictionary, else default.

items

def items(
    ...
)

D.items() -> a set-like object providing a view on D's items

keys

def keys(
    ...
)

D.keys() -> a set-like object providing a view on D's keys

pop

def pop(
    ...
)

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem

def popitem(
    self,
    /
)

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault

def setdefault(
    self,
    key,
    default=None,
    /
)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update

def update(
    ...
)

D.update([E, ]**F) -> None. Update D from dict/iterable E and F.

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values

def values(
    ...
)

D.values() -> an object providing a view on D's values

MissingDependencyError

class MissingDependencyError(
    /,
    *args,
    **kwargs
)

Raised when a missing optional dependency is detected.

Ancestors (in MRO)

builtins.RuntimeError
builtins.Exception
builtins.BaseException

Class variables

args

Methods

add_note

def add_note(
    ...
)

Exception.add_note(note) --

add a note to the exception

with_traceback

def with_traceback(
    ...
)

Exception.with_traceback(tb) --

set self.traceback to tb and return self.

NamedDataset

class NamedDataset(
    /,
    *args,
    **kwargs
)

Dataset abstraction.

Dataset is an Iterable[DatasetItem], and has the following attributes:

name (str): Dataset name
split (str): Data split ("test" or "train")
labels (dict): Dictionary of dataset labels

Ancestors (in MRO)

collections.abc.Sized
collections.abc.Iterable

Descendants

redlite.dataset.StripSystemDataset
redlite.dataset._load.ValidatingDataset
redlite.dataset._load.TransformingDataset
redlite.dataset.memory_dataset._MemoryDataset

NamedMetric

class NamedMetric(
    name: str,
    engine: collections.abc.Callable[[str, str], float]
)

Metric abstraction.

Metric is a Callable that have name attribute.

name (str): Metric name
engine: Function that will be called to compute the score.

Sample usage:

def engine(expected: str, actual: str) -> float:
    if expected == actual:
        return 1.0
    return 0.0

hit_metric = NamedMetric('hit_metric', engine)

Descendants

redlite.metric.MatchMetric
redlite.metric.RandomMetric
redlite.metric.RejectionMetric
redlite.metric.BestOfMetric
redlite.metric.bleu.BleuMetric
redlite.metric.bleu.BleuCJKMetric
redlite.metric.f1.F1Metric
redlite.metric.livecodebench.LiveCodeBenchMetric
redlite.metric.math.BoxedMathMetric
redlite.metric.math.MathMetric
redlite.metric.rouge.RougeMetric
redlite.metric.rouge.RougeCJKMetric

NamedModel

class NamedModel(
    name: str,
    engine: collections.abc.Callable[[list[redlite._core.Message]], str]
)

Model abstraction.

Model is a Callable object that has name attribute. Given an input Messages it should return the response string.

name (str): Name of the model.
engine (Callable[[list[Messages]], str]): A function that computes model prediction from messages.

Sample usage:

def parrot_engine(messages: list[Message]) -> str:
    return messages[-1]['content']

parrot_model = NamedModel('parrot', parrot_engine)

Descendants

redlite.model.IgnoreSystemModel
redlite.model.MakeSystemModel
redlite.model.ConvertSystemToUserModel
redlite.model.RemoveThinking
redlite.model.CannedModel
redlite.model.ParrotModel
redlite.model.ThrottleModel
redlite.model.ModerationModel
redlite.model.anthropic_model.AnthropicModel
redlite.model.aws_bedrock_model.AwsBedrockModel
redlite.model.gemini_model.GeminiModel
redlite.model.hf_model.HFModel
redlite.model.llamacpp_model.LlamaCppModel
redlite.model.openai_model.OpenAIModel

Run

class Run(
    /,
    *args,
    **kwargs
)

Run metadata

Ancestors (in MRO)

builtins.dict

Methods

clear

def clear(
    ...
)

D.clear() -> None. Remove all items from D.

copy

def copy(
    ...
)

D.copy() -> a shallow copy of D

fromkeys

def fromkeys(
    iterable,
    value=None,
    /
)

Create a new dictionary with keys from iterable and values set to value.

get

def get(
    self,
    key,
    default=None,
    /
)

Return the value for key if key is in the dictionary, else default.

items

def items(
    ...
)

D.items() -> a set-like object providing a view on D's items

keys

def keys(
    ...
)

D.keys() -> a set-like object providing a view on D's keys

pop

def pop(
    ...
)

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem

def popitem(
    self,
    /
)

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault

def setdefault(
    self,
    key,
    default=None,
    /
)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update

def update(
    ...
)

D.update([E, ]**F) -> None. Update D from dict/iterable E and F.

If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values

def values(
    ...
)

D.values() -> an object providing a view on D's values