01
LLM Evaluation
2026
16K
context sweeps · 5 adversarial families
Long-Context Adversarial Eval Harness
- Built a
uv-based evaluation harness for source-grounded long-context fact QA, shaped after RULER, BABILong, and LongBench: configurable 2K/8K/16K length buckets and early/middle/late evidence positions. - Hand-verified 60-case synthetic dataset across five adversarial families (single fact, multi-key distractor, multi-value, multi-hop, misleading fact pair), with an audit log checking every answer against an exact source span.
- Forced strict JSON with cited evidence spans so hallucinations are auditable; reports accuracy, schema validity, evidence-span presence, and calibration (ECE with reliability diagrams) per family and bucket.
- Ran GPU inference on Modal with vLLM and added a regression command that fails CI when a previously passing case breaks; a 0.5B Qwen2.5 model scored 0.167 accuracy at up to 16K, exposing sharp long-context degradation.
vLLM · Modal · uv · pytest
Code