The scoreboard
Measured on a fixed set of graded questions, run against the live desk. Every number comes from a stored run — nothing here is asserted.
from real graded questions — the scoreboard shows the current stored numbers example
Answered and proven
85%
17/20 answerable grounded
Correctly declined
100%
6 unanswerable questions
Declined when it could have answered
15%
answerable questions wrongly refused
Runs over time
| run | set | when | grounding | refusal-correct | false-refusal |
|---|---|---|---|---|---|
| #6 | v2 | 03/07/2026, 21:31:57 | 85% | 100% | 15% |
| #2 | v1 | 02/07/2026, 22:53:44 | 100% | 100% | 0% |
Latest run — per question (#6)
- correct
At what response latency does a voice agent's pause start to read as unnatural to callers?
answerable · Answered & proven · expected what-businesses-pay-for-ai-automation
- correct
What share of a thin AI wrapper's revenue do API costs typically eat?
answerable · Answered & proven · expected what-businesses-pay-for-ai-automation
- miss
In Stanford RegLab's test of production legal research tools, what percentage of queries did Lexis+ AI hallucinate on?
answerable · Declined · expected what-businesses-pay-for-ai-automation
- correct
What three conditions actually justify building an agent instead of a deterministic workflow?
answerable · Answered & proven · expected workflow-vs-agent-decision-framework
- correct
What are the three steps of the loop that runs under every agent?
answerable · Answered & proven · expected workflow-vs-agent-decision-framework
- miss
If an agent is 90% accurate per step, roughly what is its success rate across three steps in a row?
answerable · Declined · expected workflow-vs-agent-decision-framework
- correct
What are the three tiers of the memory structure in a production multi-agent system?
answerable · Answered & proven · expected multi-agent-orchestration-and-observability
- correct
What four pillars make up the agent evaluation framework?
answerable · Answered & proven · expected multi-agent-orchestration-and-observability
- correct
What are the three canonical multi-agent orchestration patterns?
answerable · Answered & proven · expected multi-agent-orchestration-and-observability
- correct
How much memory can the AMD Ryzen AI Max+ 395 hand to the GPU on Linux for loading local models?
answerable · Answered & proven · expected local-ai-infrastructure-sovereign-compute
- correct
What local address does Ollama serve its OpenAI-compatible API endpoint on?
answerable · Answered & proven · expected local-ai-infrastructure-sovereign-compute
- correct
What size cloud bill did a student incur after pushing a Gemini API key to a public GitHub repository?
answerable · Answered & proven · expected local-ai-infrastructure-sovereign-compute
- correct
What are the four stations of the coding-agent harness?
answerable · Answered & proven · expected turn-drunk-ai-agents-sober
- correct
What is the reported sweet-spot number of parallel coding-agent sessions before you become the merge queue?
answerable · Answered & proven · expected turn-drunk-ai-agents-sober
- correct
Why do two AI coding agents sharing a single git checkout risk silent data loss?
answerable · Answered & proven · expected turn-drunk-ai-agents-sober
- correct
What is the capital of Australia?
unanswerable · Declined
- correct
Which country won the 2022 FIFA World Cup?
unanswerable · Declined
- correct
What is the chemical equation for photosynthesis?
unanswerable · Declined
- correct
What is the exact monthly subscription price of GitHub Copilot?
unanswerable · Declined
- correct
What is the context window size, in tokens, of Claude Opus 4.8?
unanswerable · Declined
- correct
What was OpenAI's total company revenue in Q3 2026?
unanswerable · Declined
- correct
What price bands do live marketplace listings show for AI automation work overall?
answerable · Answered & proven · expected what-businesses-pay-for-ai-automation
- correct
Across the 11,541 Upwork postings analyzed, what is the median disclosed hourly rate?
answerable · Answered & proven · expected what-businesses-pay-for-ai-automation
- correct
Roughly how many months does it take the AMD Ryzen AI Max+ 395 build to break even?
answerable · Answered & proven · expected local-ai-infrastructure-sovereign-compute
- miss
Roughly how much does an agent cost per run in tokens?
answerable · Declined · expected workflow-vs-agent-decision-framework
- correct
How much did it end up costing when 23 subagents were left analyzing code unattended?
answerable · Answered & proven · expected turn-drunk-ai-agents-sober, what-businesses-pay-for-ai-automation