Explainers· Last updated 2026-09-21

Offline evaluation for Jev decisions

Offline evaluation means: freeze the question pack, pin the model id, replay a labeled set, and score your code’s act/review/abstain — not a chat transcript.

Jev (TypeSafe System One) is a decision model. TypeSafe’s public “workflow evals” compare Jev to wrapped frontier LLMs on their graphs. That is a vendor story (including 193.6× / 444.6× claims they call high-end). This page is how you run a harness. Unofficial. Confirm APIs on docs.typesafe.ai. No keys from us.

What TypeSafe already measured (do not restate as yours)

• Launch post: workflow evals assume a correct compute graph; reference probabilities from large named LLMs; Jev claimed on the Pareto frontier for ~2 orders of magnitude.
• They warn of author bias (capabilities team wrote workflows) and wrapper overhead on the LLM side.
• Cookbook lifts (rerank top-1 5%→18%, parallel-questions 12.2× / 10.0×) stay on those cookbook pages.

If you need those figures, click through. We will not copy them into a fake leaderboard.

Minimal harness

1. Gold file — JSONL: state, expected choice / score bucket / noul ≥ τ, plus the action your product should take.
2. Pin jev-1.13.0 (or the id you will ship). Log response.model.
3. Freeze instructions and criteria in one module (TypeSafe agent-skill advice).
4. Ask the same questions your production path asks — including speculative ones you later ignore.
5. Score two layers:
   • Model layer: argmax vs gold; Brier / ECE on probabilities if you have enough rows.
   • Policy layer: did route() match gold action? This is what pages users.
6. Slice by length, language, and “adversarial / injected instruction” (jaggedness: state is not hostile by default).
7. Do not optimize the harness until the graph matches production (TypeSafe’s own warning about harness engineering).

PIN = "jev-1.13.0"
# client = TypeSafeClient(model=PIN)

def evaluate_row(row, client, route):
    r = client.system_one(state=row["state"], questions=PRODUCTION_QUESTIONS)
    pred = route(r)  # your thresholds
    return {
        "model": r.model,
        "ok_action": pred == row["gold_action"],
        "ok_choice": r.answers["intent"].choice == row.get("gold_intent"),
        "confidence": getattr(r.answers["intent"], "confidence", None),
        "input_tokens": r.usage.input_tokens,
    }

Plot confidence vs accuracy (official “route on uncertainty” guidance). Start conservative floors; move them only with this file.

What not to put in the harness

• Arithmetic or date order questions — jaggedness says keep those in code.
• Generation quality (BLEU, “helpful tone”) — Jev does not write the reply.
• A single 200-way Choice you never ship — use the cascade you will ship (classification comparison).

FAQ

Can I use TypeSafe workflow evals as my offline set? Only if they publish the exact payloads you need. Prefer your tickets.

Noul gold? Store a yes/no plus a τ you will ship. Report both Brier and thresholded accuracy.

Latency in the same job? Yes — see decision latency. Separate success p95 from retry-inflated means.

Limits

No invented 193x. Schema-safe ≠ correct. Hub: Explainers. Sibling: implementation checklist.