Acuityio Benchmark Dataset v1.0

1,000 real-world factual claims evaluated across 5 frontier models with verbatim rationales.

Dataset Format (JSONL)

JSONL Schema Preview

{"id": "claim_001", "claim": "Einstein won the 1921 Nobel Prize for relativity.", "models": {"gpt4o": {"verdict": "False", "confidence": 10}, "claude35": {"verdict": "False", "confidence": 10}, "gemini15": {"verdict": "False", "confidence": 9}}, "consensus_verdict": "False", "disagreement_degree": "none"}
{"id": "claim_002", "claim": "Failure to give Miranda warnings invalidates an arrest.", "models": {"gpt4o": {"verdict": "True", "confidence": 9}, "claude35": {"verdict": "True", "confidence": 9}, "llama405b": {"verdict": "Mostly False", "confidence": 8}}, "consensus_verdict": "True", "disagreement_degree": "moderate"}