Open· Last updated 2026-09-21

Jev testing guide

Testing Jev is not screenshotting Playground. It is frozen questions + pinned model + gold cases + assertions on actions. Date: treat this page as 2026-09-21 methodology.

Independent. Pair with evaluation guide. docs.typesafe.ai.

Independent angle (cover, do not clone)

Win delta (review-hands-on): reproducible eval/test notes with dates; methodology > opinion blog. Not a clone of a “Jev review” URL.

Layers

1. Contract tests — request JSON schema, door model id, no boolean on first-party.
2. Unit tests of your policy — given stubbed answers, action enum is correct.
3. Golden network tests — small recorded set against pinned jev-1.13.0 (flakes possible; quarantine, don’t ignore).
4. Adversarial cases — injection, contradiction, huge state (expect degradation).

What not to assert

Bit-identical probabilities forever. Exact vendor latency. “Cannot be wrong.”

Dated lab note template

date: 2026-09-21
model: jev-1.13.0
door: first-party HTTP
question_sha: <git>
n: 80 gold rows
metrics: auto_precision, review_rate, 422_count
notes: English tickets only; no images

Fill numbers from your run. Do not paste ours — we did not run your set.

Flake policy

Golden calls against a hosted model will flake (429, 529, rare answer flips). Cap retries. Quarantine known-soft cases. Never “sleep and hope” in CI.

Adversarial fixtures belong in the repo: injection, date-math bait, hex colors vs names (jaggedness). Expect the model to be weak where TypeSafe said it is weak.

What this page does not claim

• No pretended hands-on leaderboard.

FAQ

CI without keys? Mock the client in PRs; nightly job with a secret for goldens.

SDK retries in tests? Disable or bound them or your suite becomes a load test.

Disclaimer

This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.

Open-cluster pages are independent field-guide notes. Replicas and third-party interfaces mentioned anywhere on jev.pro are not Jev and not endorsed. Hub: Open. Siblings: evaluation guide, offline evaluation, best practices. Canonical: https://docs.typesafe.ai.