API reference¶

What import evaltrack exports: the calls a marked test makes, the models a recorded run is made of, the repository read path, and the exceptions. The marker itself is not a Python callable. Its keyword arguments are in Configuration › Marker keyword arguments.

`evaltrack`¶

evaltrack records, gates and tracks LLM eval runs from pytest.

version `module-attribute` ¶

__version__ = importlib.metadata.version('evaltrack')

The installed evaltrack version, as PyPI knows it.

AttemptErrorRecord ¶

Bases: BaseModel

Something that raised while producing or judging one attempt.

Attributes:

Name	Type	Description
`message`	`UserStr`	The exception's type and message, without a traceback.
`evaluator`	`str \| None`	The evaluator that raised, or None when the task raised.

AttemptRecord ¶

Bases: BaseModel

One attempt at a case.

Attributes:

Name	Type	Description
`outcome`	`AttemptOutcome`	`errored` when the task or an evaluator raised. The attempt reached no verdict.
`task_duration`	`FiniteFloat \| None`	Seconds the task took, or None.
`output`	`UserValue`	The task's output. None when the task raised.
`results`	`dict[str, EvaluatorResult]`	What each evaluator returned, keyed by result name, with the bar that applied already resolved onto it.
`errors`	`list[AttemptErrorRecord]`	Everything that raised, each with the evaluator that raised it, or no evaluator when the task did. Empty on a clean attempt.
`details`	`dict[str, UserValue]`	Runner-specific data, stored as produced and never interpreted.

CaseRecord ¶

Bases: BaseModel

Everything recorded for one case in one run.

inputs, expected_output and metadata describe the case rather than any one attempt at it, so they are taken from the first attempt and the rest have to agree with it.

Attributes:

Name	Type	Description
`inputs`	`UserValue`	What the task was given, as the runner reported it.
`expected_output`	`UserValue`	What the case expected, or None when it declares none.
`metadata`	`UserValue`	The case's own metadata, as the runner reported it.
`attempts`	`list[AttemptRecord]`	Every attempt at the case, earliest first.
`passed_attempts`	`int`	Clean attempts that passed the gate.
`clean_attempts`	`int`	Attempts that did not error.
`errored_attempts`	`int`	Attempts whose task or an evaluator raised.
`outcome`	`CaseOutcome`	`errored` when any attempt errored. Otherwise every attempt must pass when `repeats` is above 1, and one passing attempt is enough when it is not.

CorruptRecordError ¶

Bases: EvaltrackError

Stored bytes that cannot be read as the record they are keyed as, a run or a reflog. Raised only when a caller asked for that record.

EvalDefinitionError ¶

Bases: EvaltrackError

The eval as defined cannot produce a verdict. Some of this shows only after the eval ran, but none of it depends on what the model answered, so running the eval again cannot change it.

EvalExecutionError ¶

Bases: EvaltrackError

A task or evaluator raised.

EvalRound ¶

Bases: BaseModel

One batch of cases, put through the runner once. Build one directly to record an eval from a runner that has no translator.

Attributes:

Name	Type	Description
`runner`	`RunnerInfo`	Which eval runner produced the batch, and at what version.
`attempts`	`list[RoundAttempt]`	Every attempt in the batch, in the runner's order. Repetitions of one case are separate entries sharing a `case_id`.
`errors`	`list[RoundErrorRecord]`	What failed outside any single attempt.
`details`	`dict[str, UserValue]`	Runner-specific data, stored as produced and never interpreted.

EvaltrackError ¶

Bases: Exception

Base class of every public evaltrack exception.

EvaluatorInfo ¶

Bases: BaseModel

Which evaluator produced a result, and how it was configured.

Attributes:

Name	Type	Description
`name`	`str`	The evaluator's own name, not always the result's key.
`arguments`	`UserValue`	What the evaluator was built with, as the runner reports it. A runner can report only the settings the caller passed, and not the defaults. None when the runner reports no configuration. What the evaluator answered goes on the result, never here.

EvaluatorResult ¶

Bases: BaseModel

One evaluator's result for one attempt.

A score carries verdict: None until a bar reaches it, so the marker's score_bars can still decide.

Attributes:

Name	Type	Description
`value`	`ResultValue`	What the evaluator returned.
`verdict`	`bool \| None`	Whether the result passes. Derived, never set directly.
`bar`	`FiniteFloat \| None`	The number `value` must reach. Derived, the marker's bar over the runner's, and never set directly.
`runner_bar`	`FiniteFloat \| None`	The bar the runner set, whether or not the marker's replaced it. None when the runner set none.
`marker_bar`	`FiniteFloat \| None`	The bar the marker declared for this result in `score_bars`. None when it declared none.
`reason`	`UserStr \| None`	Human-readable explanation, or None.
`evaluator`	`EvaluatorInfo`	Which evaluator produced the value, and how it was configured.
`details`	`dict[str, UserValue]`	Runner-specific data about this result, stored as produced and never interpreted.

is_assertion `property` ¶

is_assertion: bool

Whether value is its own verdict, needing no bar.

is_score `property` ¶

is_score: bool

Whether a bar can grade value. An assertion answers for itself, and True >= 0.5 holds, so a bar must never reach one.

InvalidIdentifierError ¶

Bases: EvaltrackError

A ref name or run id that breaks the naming rules.

MarkerSettings ¶

Bases: BaseModel

evaltrack metadata for a test's recorded eval.

Attributes:

Name	Type	Description
`score_bars`	`dict[str, FiniteFloat]`	Per-case score bars, keyed by score name.
`repeats`	`int`	Attempts each case needed, all of which must pass.
`flake_reruns`	`int`	The marker's budget of extra rounds while a case was still unproven.
`reliability_target`	`float \| None`	Cross-run pass-rate target for reporting. Never fails a test.
`eval_version`	`str \| None`	Version label to bump whenever anything that could move the result changes: the cases, the evaluators, the bars, the judge or the system under test. Runs sharing it are the ones compared with one another, so a bump starts a case's history over.

RecordedTest ¶

Bases: BaseModel

The recorded eval and pytest outcome for one test.

Attributes:

Name	Type	Description
`cases`	`dict[str, CaseRecord]`	Per-case results keyed by case id.
`errors`	`list[RoundErrorRecord]`	What failed outside any single case, every round's in the order the rounds ran. A rerun appends to this rather than replacing it, so a round that failed is still readable once a later one has run.
`marker`	`MarkerSettings \| None`	What the marker declared. None when the test recorded no eval.
`runner`	`RunnerInfo \| None`	Which eval runner produced the eval, at what version. None when the test recorded no eval.
`raw_results`	`list[UserValue]`	The runner's own result objects, one per round. Empty when keep_raw_results is false.
`outcome`	`TestOutcome \| None`	pytest's outcome for the test, which is the authoritative one. It can disagree with every case below it, since a test whose eval passed can still fail later in its body.
`docstring`	`str \| None`	The test function's docstring, when it has one.
`test_file`	`str \| None`	The test file's path from the nodeid, so relative to pytest's rootdir.
`details`	`dict[str, UserValue]`	Runner-specific data for the eval as a whole.

ReflogEntry ¶

Bases: BaseModel

One entry in a ref's reflog. It names the run the ref moved to, not the ref.

Attributes:

Name	Type	Description
`run_id`	`RunId`	The run the ref moved to.
`moved_at`	`AwareDatetime`	When the move was recorded.
`commit`	`str \| None`	The commit recorded for this move, if any.
`pr`	`int \| None`	The pull request number for the move, if any.
`title`	`str \| None`	The pull request title for the move, if any.

RepositoryUnavailableError ¶

Bases: EvaltrackError

The storage cannot serve the request. The cause is the environment, not a bug and not a missing run or ref. No storage SDK exception type escapes.

RoundAttempt ¶

Bases: BaseModel

One attempt at one case.

Attributes:

Name	Type	Description
`case_id`	`str`	The case this is an attempt at. Never empty, since it is the key attempts are grouped by.
`inputs`	`UserValue`	What the task was given, as the runner reported it.
`expected_output`	`UserValue`	What the case expected, or None when it declares none.
`metadata`	`UserValue`	The case's own metadata, as the runner reported it.
`output`	`UserValue`	What the task produced. None when the task raised.
`results`	`dict[str, EvaluatorResult]`	What each evaluator returned, keyed by result name.
`task_duration`	`FiniteFloat \| None`	Seconds the task took, or None.
`errors`	`list[AttemptErrorRecord]`	What raised while producing or judging this attempt. An attempt with any error never passes.
`details`	`dict[str, UserValue]`	Runner-specific data, stored as produced and never interpreted.

passed `property` ¶

passed: bool

Whether the attempt is clean and no result failed.

A verdict of None is a score the marker's bars have not reached yet. The round is refused if one survives them, so by the time anything asks this, every result has answered.

RoundErrorRecord ¶

Bases: BaseModel

A failure that belongs to the whole round rather than to one attempt.

Attributes:

Name	Type	Description
`name`	`str`	What the runner calls this failure.
`message`	`UserStr`	The exception's type and message, without a traceback.

RunRecord ¶

Bases: BaseModel

An immutable recorded run.

Attributes:

Name	Type	Description
`run_schema_version`	`int`	`RUN_SCHEMA_VERSION` at record time. Required, so bytes without a stamp are treated as corrupt.
`id`	`RunId`	ULID assigned at record time, and the storage key.
`created_at`	`AwareDatetime`	When the run was recorded. Timezone-aware.
`recorded_by`	`RecordedBy`	Versions of the tools that produced the run.
`commit`	`str \| None`	The commit the run was evaluated at, or None when unknown.
`worktree_dirty`	`bool \| None`	None when unknown.
`labels`	`dict[str, str]`	Arbitrary key-value metadata.
`tests`	`dict[str, RecordedTest]`	Everything recorded per test, keyed by pytest nodeid.

RunRepository ¶

RunRepository(store: ObjectStore)

A repository of immutable runs and mutable refs, kept in an ObjectStore. Only the store differs between backends.

Names and ids are checked, never corrected. A run id is a ULID and a ref name is lowercase, so every comparison over a stored name or id is an exact match.

Any method can raise RepositoryUnavailableError when the storage cannot serve the request.

delete_ref_unchecked ¶

delete_ref_unchecked(name: str) -> None

Delete a ref by deleting its reflog. A name that does not exist is a no-op.

This is the storage primitive, not the way to remove a ref. It skips every guard delete_ref_and_orphaned_runs applies, baseline included. Runs that only this ref reached stay stored, and nothing reclaims them.

It reads no history, so it can still drop a ref whose reflog no longer parses.

delete_run_unchecked ¶

delete_run_unchecked(run_id: str) -> str | None

Delete one run. The id if deleted, None if it did not exist.

This is the storage primitive. It does not ask whether a ref still reaches the run, so a delete here can leave a ref that points at nothing. delete_run_if_unreferenced is the way to remove a run.

get_ref ¶

get_ref(name: str) -> ReflogEntry | None

The ref's current target (its newest reflog entry), or None if the ref does not exist.

Raises:

Type	Description
`CorruptRecordError`	when an entry does not parse.

get_reflog ¶

get_reflog(name: str) -> Iterator[ReflogEntry]

Yield a ref's whole history, oldest-first. Nothing if the ref does not exist.

Raises:

Type	Description
`CorruptRecordError`	when an entry does not parse.

get_run_summary ¶

get_run_summary(run_id: str) -> RunSummary | None

One run's summary, or None if absent. One read, not a scan.

Raises:

Type	Description
`CorruptRecordError`	when the stored run does not parse.

list_refs ¶

list_refs() -> Iterator[str]

Enumerate ref names as stored, including a key that is not a valid ref name, because retention needs the whole set. Reading such a key raises InvalidIdentifierError.

list_runs ¶

list_runs() -> Iterator[RunSummary]

Yield one summary per run, newest-first. A run recorded under a schema this version does not read is skipped.

Raises:

Type	Description
`CorruptRecordError`	when a stored run does not parse.

load_run ¶

load_run(run_id: str) -> RunRecord | None

Load one run in full, or None if absent.

Raises:

Type	Description
`CorruptRecordError`	when the stored run does not parse.
`UnsupportedSchemaError`	when the run was recorded under a schema this version does not read. The stored run is intact.

move_ref ¶

move_ref(
    name: str,
    run_id: str,
    *,
    commit: str | None = None,
    pr: int | None = None,
    title: str | None = None,
) -> ReflogEntry

Repoint a ref and append a reflog entry, atomically. The ref is created if absent.

Idempotent, so a move to the run the ref already points at appends no entry, and records none of the metadata passed with it.

Returns:

Type	Description
`ReflogEntry`	The ref's tip entry after the move.

Raises:

Type	Description
`InvalidIdentifierError`	when no ref can be stored under `name`, or `run_id` is not a valid run id.

save_run ¶

save_run(run: RunRecord) -> None

Save a run. A save under an existing run.id replaces the stored run. Content is not compared, so an id must never be reused for different content.

Raises:

Type	Description
`RepositoryUnavailableError`	when the storage cannot serve the write.

tail_reflog ¶

tail_reflog(name: str, n: int) -> list[ReflogEntry]

The last n entries, newest-first. Empty when the ref does not exist or n <= 0.

Raises:

Type	Description
`CorruptRecordError`	when an entry does not parse.

validate_ref_name ¶

validate_ref_name(name: str) -> str

Check that a ref name is storable, without touching storage. Rejects exactly the names move_ref rejects.

Raises:

Type	Description
`InvalidIdentifierError`	when this repository cannot store a ref under it.

verify_available ¶

verify_available() -> None

Cheaply check that the backing storage can serve requests.

Raises:

Type	Description
`ValueError`	when the storage is misconfigured.
`RepositoryUnavailableError`	when the storage cannot be reached.

verify_writable ¶

verify_writable() -> None

Cheaply check, without network I/O, that a save can succeed.

Raises:

Type	Description
`RepositoryUnavailableError`	when a save cannot succeed.

RunSummary ¶

Bases: BaseModel

Small projection of RunRecord for list views. Never holds reports.

Attributes:

Name	Type	Description
`id`	`RunId`	The run's ULID.
`created_at`	`AwareDatetime`	When the run was recorded.
`commit`	`str \| None`	The commit the run was evaluated at, if known.
`worktree_dirty`	`bool \| None`	Whether the working tree was dirty at eval time, if known.
`labels`	`dict[str, str]`	The run's labels.
`tests_total`	`int`	How many tests the run recorded.
`tests_failed`	`int`	How many tests failed or errored.

RunnerInfo ¶

Bases: BaseModel

Which eval runner produced a result, and at what version. version is None when the runner does not report one.

TranslatorNotFoundError ¶

Bases: EvaltrackError

No registered translator handles a result.

UnsupportedSchemaError ¶

Bases: EvaltrackError

A run stamped with a stored format this evaltrack does not read. The fix is to use the evaltrack that wrote it, not to delete the run.

open_repository ¶

open_repository(location: str) -> RunRepository

Open a RunRepository for location, a directory path, or a URL with a scheme. This does not check that the storage exists or can be reached, verify_available does.

Raises:

Type	Description
`ValueError`	for a URL whose scheme no backend claims, or one written without its `://`.
`ImportError`	when a built-in scheme's optional dependency is absent.

repeats ¶

repeats() -> int

The attempts per case the marker requires, for an eval that repeats them itself.

Pass it as pydantic-evals repeat=. The eval must then produce exactly this many attempts. Returns 1 when the marker requires nothing.

Raises:

Type	Description
`EvalDefinitionError`	when no marked test is running.

run ¶

run(
    evaluate: Callable[P, Any],
    /,
    *args: args,
    **kwargs: kwargs,
) -> None

Run an eval through evaltrack, and add rounds as the marker requires.

Call this from a test marked @pytest.mark.evaltrack(...). Every round is recorded, and the test fails when a case fails. docs/marker.md says how many rounds the marker asks for, and which attempts must pass.

Parameters:

Name	Type	Description	Default
`evaluate`	`Callable[P, Any]`	Runs the eval and returns whatever the runner produces. It is called once per round, so it must run the eval each time, over the same dataset. A round that adds, drops or renames a case is refused.	required
`*args`	`args`	Passed to `evaluate` on every round.	`()`
`**kwargs`	`kwargs`	Passed to `evaluate` on every round.	`{}`

Raises:

Type	Description
`EvalDefinitionError`	in any of these cases. - No marked test is running. - `evaluate` is a result, not something to call. - `evaluate` must be awaited. - The eval has no cases. - A round does not agree with the first round, or with the marker.
`TranslatorNotFoundError`	when no registered translator takes what the eval returned.
`EvalExecutionError`	when a task or evaluator raised. An error is never retried.
`AssertionError`	when a case failed the gate.

For an eval whose entry point is a coroutine, use run_async.

run_async `async` ¶

run_async(
    evaluate: Callable[P, Awaitable[Any]],
    /,
    *args: args,
    **kwargs: kwargs,
) -> None

run for an eval that must be awaited, such as pydantic-evals' Dataset.evaluate. Same arguments and exceptions as run.

The round loop below is the same as the loop in run, with await added. If you change one loop, change the other.

Extension surface¶

A translator turns an eval runner's own result into an EvalRound. Registering one is how another runner joins pydantic-evals and DeepEval. See Translators.

register ¶

register(
    name: str,
    *,
    native_types: tuple[str, ...],
    load: Callable[[], Translator],
) -> None

Register a translator. A later register under the same name replaces the earlier one. That is also how to replace a shipped translator.

Parameters:

Name	Type	Description	Default
`name`	`str`	The registry key, and what refusals here call the translator. What a round reports in `EvalRound.runner` comes from the translator itself, not from this.	required
`native_types`	`tuple[str, ...]`	Dotted paths of the result types this translator handles, compared as text. A claim on a subclass of a claimed type wins for results of that subclass.	required
`load`	`Callable[[], Translator]`	Builds the translator. Called at most once per process, so an expensive translator is built once however many rounds look it up.	required

Raises:

Type	Description
`ValueError`	when `native_types` is empty, or when another name already claims one of them.

Translator ¶

Bases: Protocol

Turns one eval runner's results into evaltrack's model.

translate ¶

translate(result: Any) -> EvalRound

Build an EvalRound from the runner's own result object.

API reference¶

evaltrack¶

__version__ module-attribute ¶

AttemptErrorRecord ¶

AttemptRecord ¶

CaseRecord ¶

CorruptRecordError ¶

EvalDefinitionError ¶

EvalExecutionError ¶

EvalRound ¶

EvaltrackError ¶

EvaluatorInfo ¶

EvaluatorResult ¶

is_assertion property ¶

is_score property ¶

InvalidIdentifierError ¶

MarkerSettings ¶

RecordedTest ¶

ReflogEntry ¶

RepositoryUnavailableError ¶

RoundAttempt ¶

passed property ¶

RoundErrorRecord ¶

RunRecord ¶

RunRepository ¶

delete_ref_unchecked ¶

delete_run_unchecked ¶

get_ref ¶

get_reflog ¶

get_run_summary ¶

list_refs ¶

list_runs ¶

load_run ¶

move_ref ¶

save_run ¶

tail_reflog ¶

validate_ref_name ¶

verify_available ¶

verify_writable ¶

RunSummary ¶

RunnerInfo ¶

TranslatorNotFoundError ¶

UnsupportedSchemaError ¶

open_repository ¶

repeats ¶

run ¶

run_async async ¶

Extension surface¶

register ¶

Translator ¶

translate ¶

`evaltrack`¶

version `module-attribute` ¶

is_assertion `property` ¶

is_score `property` ¶

passed `property` ¶

run_async `async` ¶