RAG failure modes with Jev
Retrievers hope. After retrieval, Jev marks relevance, contradiction, or injection; code keeps, flags, or drops passages before a generator sees them.
This unofficial page is the failure modes slice of the RAG passage decisions pack. Intent: apply the Jev (TypeSafe System One) decision model to RAG passage decisions failure modes. Primary search language: RAG Jev failure modes. Confirm patterns on docs.typesafe.ai. This site does not sell, issue, or proxy TypeSafe keys. Use a credential you already have from the console or a documented gateway.
Independent angle (cover ≠ clone): Filter/rerank recipes with failure modes; cite vs generate boundary. We cover the intent, not a rival rerank-passages-score URL tree.
RAG use-case context
RAG passage decisions breaks in product-specific ways. This page lists those modes so you can write tests — not a generic “AI can be wrong” essay, and not a rival limitations-page clone.
Hub: Classifying RAG passages. Compare, when the other tool is the real job: RAG pipelines.
Failure Modes inputs
Many failures start as contract violations (distractors, missing query + retrieved passage text). Canonical shape:
{
"query": "What is the refund window for pro plans?",
"passage": { "id": "doc-88#p3", "text": "Pro subscribers may request a refund within 14 days." },
"corpus": { "trust": "internal_kb" }
}Decision signals and actions
- Retrieval-only search still returns junk; Jev is a post-filter, not a magic index (compare).
- Large noisy state is a documented
jev-1.13weakness — filter in code first. - A relevant Noul is not a citation. Citation is a different question (see the citation pack).
- We will not restate unofficial cookbook accuracy percentages — read TypeSafe’s page.
- Schema-safe
relevant≠ the passage is factually true.
HTTP vs application:
| You see | Class | RAG move |
|---|---|---|
| 401 / 422 / 429 / 529 | Documented HTTP | Fix key/body or back off — errors |
| 200 + flat confidence or Noul ≈ 0.5 | Low confidence | Hold; do not feed the passage to the customer-facing answerer |
| Empty gather | Missing evidence | Skip Jev or ask “is enough information present?” |
Do not treat a Noul of 0.5 as a “medium” RAG passage decisions score — it means yes and no are equally likely. Conjunctions stay in your code.
Guardrails and escalation
Fail closed: do not feed the passage to the customer-facing answerer. Schema-safe answers are not factual correctness. TypeSafe’s confidence-gated examples use a lower bar for recoverable reads than for irreversible actions. Those numbers are illustrations. For RAG passage decisions, treat feed_to_answerer as the high bar (showing a passage to a customer-facing answerer). Tune on labels — see offline evaluation.
Evaluation and rollout notes
Your canary set should include each bullet above.
- Precision@k of kept passages vs your labels
- Generator groundedness after the filter (your harness)
- Drop rate on an injection-canary set
Pin jev-1.13.0 (the versioned id) after you fit thresholds. jev-latest and the marketing line jev-1.13 can move. Log the response model. TypeSafe’s published list price for jev-1.13 is $0.042 per million input tokens (vendor claim — confirm on the models page); output tokens are free on that same page. Unused distractors still bill as input.
Official Python and JavaScript SDKs read TYPESAFE_API_KEY and retry documented 429/529. This site does not sell, issue, or proxy TypeSafe keys. Use a credential you already have from the console or a documented gateway.
Pack map
| Slice | Page |
|---|---|
| Graph and primitives | decision workflow |
What may enter state |
input contracts |
| What to gather first | evidence collection |
| Atomic rules | policy checks |
| Act / review / abstain | confidence thresholds |
| Reviewer payload | human handoff |
| What to persist | audit trail |
| How it breaks | you are here |
| Labeled replay | evaluation |
| Shadow → canary | production rollout |
FAQ
If the API returns 200, is the decision good? 200 only means the call parsed. Low confidence, Noul ≈ 0.5, or a policy miss are application failures.
Where do official weaknesses live? TypeSafe’s jev-1.13 jaggedness note — distractors, arithmetic, adversarial content. We do not invent more.
Where is the rest of the RAG pack? Start with RAG evaluation and RAG decision workflow. Cluster hub: Use cases.
Should Jev generate the RAG answer? No. Classify or score passages; another model (or extractive code) writes. That is the cite-vs-generate boundary.
Do we publish rerank lifts? No. TypeSafe’s cookbooks may show measurements — treat those as vendor figures and re-run on your corpus.
What this page does not claim
- No invented top-1 / top-10 lifts.
- Not a vector database.
- Not official TypeSafe.
- Official TypeSafe status, or that jev.pro issues API keys.
- That a schema-constrained answer is automatically factually correct.
Disclaimer
This is an independent unofficial site and is not affiliated with TypeSafe AI; official documentation is available at https://docs.typesafe.ai.
Primary documentation: https://docs.typesafe.ai. Hub: Use cases.
Sources
Public TypeSafe or adjacent documentation only. No private claims.