Comparisons· Last updated 2026-09-21

Jev versus online experimentation: a practical comparison

Online experimentation (A/B, interleaving) estimates user-outcome lift. Jev is not an experimentation platform. It can be a treatment: variant B uses a new question pack.

Unofficial. Flags still assign buckets. docs.typesafe.ai.

Comparison scope

If jev-latest moves mid-test, your experiment is confounded. Pin both variants’ model ids and git SHAs of criteria.

Criteria that decide the architecture

Decision quality and control

A better Noul calibration that tanks conversion is a failed ship. Optimize the action outcome, not model self-love.

Integration trade-offs

Flag → variant question module → same logging schema → experiment platform. Offline harness gates who is eligible to enter the A/B.

TypeSafe’s public models page lists jev-1.13 at $0.042 per million input tokens with output tokens free — a vendor claim, not a jev.pro measurement. Confirm on the models page before you bid.

When each approach fits

Prefer Jev when

• You want to test a decision policy

Prefer Online experimentation when

• You need the experimentation product itself

Offline first, online second, flags assign, Jev executes a variant.

What this page does not claim

• No invented lift.

• Schema-safe output is not the same as factual correctness.

• No independent bake-off numbers live here.

FAQ

Can Jev pick the winning variant? That’s peeking. Use your stats stack.

Multi-armed bandits? Your code. Jev is not a bandit API.

Disclaimer

This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai. Never treat jev.pro as TypeSafe official documentation. We do not sell, issue, or proxy API keys.

Hub: Comparisons. Siblings: vs feature flags, vs offline evaluation, evaluation guide. Canonical: https://docs.typesafe.ai.