Skip to content

Module redlite.metric

Sub-modules

Classes

BestOfMetric

class BestOfMetric(
    *metrics: redlite._core.NamedMetric
)

Computes several metrics and chooses the best score.

Useful when we expect a rejection: we can use MatchMetric to detect rejection and if rejection is not expected, we compute semantic closeness with BLEU or ROUGE.

  • metrics

Ancestors (in MRO)

  • redlite._core.NamedMetric

MatchMetric

class MatchMetric(
    ignore_case=False,
    ignore_punct=False,
    strategy: Literal['exact', 'contains', 'prefix'] = 'exact'
)

Metric that checks that the actual response matches expected string.

For example, the expected response could be "Correct", but model answers "Correct, because blah blah blah...". To give model full marks for longer and verbose answer, use this metric.

This metric matches words, not strings. This means that, for example, when strategy="prefix" and expected="correct" and actual="correctness is when things work right", this metric will score 0.0.

This metric normalizes whitespace in both expected and actual, and strips any leading or trailing space there.

  • ignore_case (bool, optional) - when set to True will ignore text case. Deafult is False.

  • ignore_punct (bool, optional) - when set to True punctuation symbols will be ignored. Default is False.

  • strategy (Literal["exact", "prefix", "contains"], optional) - determines how strings are matched.

    • "exact": matches if expected and actual responses are the same
    • "prefix": matches if actual response starts with the expected words
    • "contains": matches if expected sequence of words is found found anywhere in the actual response

    Default is "exact".

Ancestors (in MRO)

  • redlite._core.NamedMetric

RandomMetric

class RandomMetric(

)

Useless metric that produces random score. May be handy for testing.

Ancestors (in MRO)

  • redlite._core.NamedMetric

RejectionMetric

class RejectionMetric(
    *,
    patterns: list[str] | None = None,
    extra_patterns: list[str] | None = None
)

Metric that asserts that LLM refuses to answer the question.

  • patterns (list[str], optional): list of user-defined RegEx patterns that match rejection answer. If not set, it will use a list of standard rejection messages, like "i am sorry, as a large language model i...".
  • extra_patterns (list[str], optional): list of extra patterns. Allows user to add new patterns to the built-in set.

Ancestors (in MRO)

  • redlite._core.NamedMetric