Jev versus offline evaluation: a practical comparison
This slug looks like “Jev versus eval.” The useful intent is: can we ship Jev instead of building an offline harness? No. Jev is the system under test. Offline evaluation is how you know whether to ship.
Unofficial. Deep how-to: offline evaluation explainer. Vendor 193×-style figures stay vendor. docs.typesafe.ai.
Comparison scope
TypeSafe publishes workflow evals and cookbook lifts. Those are their graphs. Your harness scores your actions on your gold.
Criteria that decide the architecture
| Axis | Jev (System One) | Offline evaluation (as a substitute) |
|---|---|---|
| Role | System under test | Measurement |
| Gold labels | You collect | The harness consumes |
| Pin | jev-1.13.0 |
Frozen questions + model |
| Success metric | Product action | Precision/recall/cost of those actions |
Decision quality and control
Shipping on a blog chart is how you discover your tickets are not their workflows. Score act/review/abstain, not only argmax.
Integration trade-offs
JSONL gold → pinned client → same questions as prod → two-layer score (label + policy). CI on harness diffs when criteria change.
TypeSafe’s public models page lists jev-1.13 at $0.042 per million input tokens with output tokens free — a vendor claim, not a jev.pro measurement. Confirm on the models page before you bid.
When each approach fits
Prefer Jev when
- You need a decision model (then you still eval it)
Prefer Offline evaluation (as a substitute) when
- You are choosing how to measure — pick the harness, not a rival model
Jev without eval is a demo. Eval without a model is a spreadsheet. Ship both.
What this page does not claim
-
We do not restate TypeSafe leaderboard numbers as ours.
-
Schema-safe output is not the same as factual correctness.
-
No independent bake-off numbers live here.
FAQ
Is Playground enough? For vibe checks. Not for production floors.
Online vs offline? See vs online experimentation.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.
Hub: Comparisons. Siblings: offline evaluation, vs online experimentation, evaluation guide. Canonical: https://docs.typesafe.ai.
Sources
Public TypeSafe or adjacent documentation only. No private claims.