← All topics Enterprise RAG series

08 — Grounded or Confidently Wrong: Simple Faithfulness Checks

Problem: Models invent details not supported by retrieved context. You need cheap guards before showing an answer.

In this notebook: Token overlap heuristic: flag sentences in a draft answer whose content words are missing from retrieved chunks (no LLM required).

import re
from typing import List

def content_words(s: str) -> set[str]:
    return set(re.findall(r"[a-zA-Z]{3,}", s.lower()))

def sentence_support(sentence: str, contexts: List[str], min_frac: float = 0.5) -> bool:
    sw = content_words(sentence)
    if not sw:
        return True
    ctx = " ".join(contexts).lower()
    hits = sum(1 for w in sw if w in ctx)
    return (hits / len(sw)) >= min_frac

contexts = [
    "Refund policy: enterprise customers may request a refund within 30 days of invoice.",
]
draft = (
    "Customers can get refunds within 30 days. "
    "They also receive a free lifetime license."  # unsupported
)

for sent in re.split(r"(?<=[.!?])\s+", draft.strip()):
    if not sent:
        continue
    ok = sentence_support(sent, contexts, min_frac=0.5)
    print(("OK " if ok else "FLAG"), sent)

Takeaways

• Heuristics catch obvious fabrication; pair with citation-required UX for high-risk domains.
• Stronger checks: NLI (entailment), LLM-as-judge (cost/latency), or ** abstain** when max retrieval score is low.
• Always log retrieval scores with the answer for postmortems.