Comparing models and thinking effort - With modeling

Each model was given 11 insurance-domain questions (claims, policies, premiums) and asked to answer them via two methods: Semantic Layer (MetricFlow queries) and SQL (direct SQL generation). Each model/effort combination was run multiple times to account for variance.

Note: Runs on this page use the schema with modeling (additional dbt models resolve the "too many hops" limitation, so all 11 questions are answerable by both methods). The mcp strategy is excluded from this page.

Summary

Accuracy

Latency

Cost

Tradeoffs

Ideal: top-left corner (high accuracy, low cost/latency)