Historical Comparison (2023 → 2026)
The benchmark was originally run in November 2023 to test whether LLMs could answer business questions more accurately via a Semantic Layer (MetricFlow) versus generating raw SQL directly. This page re-runs those same 11 questions with modern models to measure how much LLMs have improved over time — first in March 2026, then again in August 2026 — and shows what changes when additional dbt models are introduced to remove the "too many hops" limitation.
The section directly below leads with the latest with modeling results across both 2026 checkpoints. The remaining sections dig into the original March 2026 without modeling comparison against the 2023 baseline in more detail.
With modeling: Mar 2026 → Aug 2026
This section shows the impact of adding dbt models to the project. Two things change: the Semantic Layer gains new models that resolve the "too many hops" limitations (so all 11 questions become answerable), and the Text to SQL generator works against a richer schema. All 11 questions are included here — the too-many-hops questions are no longer a special case.
The Aug 2026 models (Claude Sonnet 5, GPT-5.6 Terra, GPT-5.6 Sol) were each swept across multiple thinking-effort levels; the table and charts below show each at its single best-performing effort, for a fair one-row-per-model comparison against the Mar 2026 models (which only have one data point each). See Comparing models and thinking effort - With modeling for the full effort breakdown.
Summary
Per-question detail
Re-running the benchmark (without modeling)
All runs in this section and the two below it use raw DDL without additional modeling — this is the original March 2026 checkpoint, shown here for the detailed 2023-vs-2026 narrative.
The first chart shows the original 2023 results per question. The two charts below re-run the same benchmark in Mar 2026 with Sonnet 4.6 and GPT-5.3 Codex respectively. The summary table at the bottom aggregates accuracy across answerable questions, too-many-hops questions, and all questions combined.
Summary
Semantic Layer: 2023 vs 2026
Isolating the Semantic Layer method to compare 2023 versus 2026 performance question by question. Note that the too-many-hops questions are included here — the Semantic Layer consistently scores 0% on those regardless of year or model, since it cannot express the required joins without additional modeling.
Text to SQL: 2023 vs 2026
Same comparison for the Text to SQL method. Unlike the Semantic Layer, Text to SQL can attempt the too-many-hops questions — it has no built-in awareness that certain joins are problematic — so results on those questions reflect whether the model happened to produce correct SQL, not whether it correctly refused.