Two Years of LLM Progress on Data Analysis

Nov 12, 2026: Ten older models, from GPT-4o to GPT-5.4, run through the same exam as today's.

At Plotly, we keep a close eye on model releases. Our agentic analytics products, Plotly Studio Desktop and Plotly Studio Embedded, and our open source libraries Plotly and Dash work with any model, and our customers look to us for recommendations.

Since September we've run every new model through DataAnalyticsBench, our exam of 43 data analysis questions. That told us which model is best today, but not how fast things are moving. So this week we went back and ran ten older models through the same exam, with the same harness and grading: OpenAI's GPT-4o, GPT-5 and GPT-5.4, Meta's Llama 3.3 70B and Llama 4 Maverick, DeepSeek V3, Google's Gemini 2.5 Pro, Moonshot's Kimi K2, and Anthropic's Claude Opus 4.1 and Claude Opus 4.6.

Together with the 47 models already on our board, that's 57 models released between November 2024 and November 2026. Two years ago, the best model we tested, GPT-4o, passed 20 of 43 questions. A year later, GPT-5 passed 35. Today, three models pass all 43, and GPT-6 Luna beats GPT-5's score for about a thirty-sixth of the cost.

The ten older models

† = open weights. Release dates are when each model became available on OpenRouter, which is how we ran all of them. GPT-4o is the November 2024 version. Costs are at OpenRouter's prices today, for one full run of the exam.

What changed

The exam has six kinds of questions, and they didn't improve at the same pace.

The straightforward analytics were mostly solved by mid-2025. These are questions with one right number, like a regional total or a ranking with ties. GPT-4o got 13 of 20, usually losing points to small SQL mistakes: comparing dates to timestamps as text, or ending a month at midnight on its last day. By August 2025, Gemini 2.5 Pro, Kimi K2 and Claude Opus 4.1 each got 18, and GPT-5 got 19.

Judgment came next. The ambiguous questions reward a model for asking what you mean, or saying which reading it chose, and the impossible ones reward it for saying the data can't answer the question. In late 2024 the models mostly guessed. By the summer of 2025, Kimi K2 and GPT-5 got all ten.

The traps took longest. Three kinds of question reward a model for checking its own work: data quality problems hidden in the warehouse, statistics traps where the obvious conclusion is wrong, and hidden patterns that only show up if you slice the data the right way. Every older model we tested scored 0 or 1 out of 3 on data quality, and none found more than two of the five hidden patterns. GPT-5.4, in March 2026, still missed every data quality problem. The models that score 42 or 43 today get almost all of these, and that's where most of the last two years of progress went.

Every lab climbed

Meta made the biggest jump. Llama 3.3 70B passed 6 questions and Llama 4 Maverick passed 4; Meta's Muse Wonder passes 41. OpenAI went from 20 with GPT-4o to 43 with GPT-6.1 Sol, and Anthropic from 28 with Claude Opus 4.1 to 43 with Claude Fable 5.2.

Open-weight models closed most of the gap. DeepSeek V3, the best open-weight model we tested from late 2024, passed 17 questions. Moonshot's Kimi K2 reached 30 in July 2025, level with Gemini 2.5 Pro. Today Kimi K3.1 Pro passes 42, one short of the best closed models (more on that in our open vs. closed post).

The price of a score keeps falling

Better models also got cheaper. When GPT-5 came out in August 2025, its score of 35 cost $4.67 a run. A year later, four open-weight models brought that price down step by step, and in September 2026 GPT-6 Luna passed 38 questions for $0.13. That's about 36 times cheaper for a better score.

For a team of 1,000 people each asking 10 questions a day, 250 working days a year, GPT-4o would cost about $339 per person per year for 20 right answers out of 43. GPT-6 Luna would cost about $7.50 for 38, and GPT-6.1 Sol about $152 for all 43.

Not every older model was cheap to run, either. Claude Opus 4.1 cost $46.61, and Gemini 2.5 Pro $47.43, more than any model on our board today except Fugu Ultra v2. Most of Gemini 2.5 Pro's bill came from two questions where it got stuck in a loop, which we cover below.

How the older models failed

Tool calls written as text. Our harness gives every model the same tools, like a SQL query tool and a chart tool, and the model has to call them. Llama 3.3 70B often wrote the call out in its reply instead, as code or JSON, so the query never ran: 16 of its 37 misses happened that way. Llama 4 Maverick did it on almost every question. On OpenRouter, Maverick has only one host that works for us, and only 2 of its 45 responses came back as real tool calls. On ten questions, Maverick went on to describe results it had never seen, with plausible-looking tables of made-up numbers. DeepSeek V3 sometimes ended its answer with raw SQL or a tool call instead of a reply, 7 times in 26 misses.

Loops and queries that never finished. Gemini 2.5 Pro made a chart on one question, then started over with "Of course. First, I need to determine the range of dates", made the same chart again, and repeated that 477 times without ever giving an answer. We stopped it after 837 model calls and $31, and it looped on one more question later. DeepSeek V3 once, and Kimi K2 twice, wrote SQL that never finished, like a self-join that matched every day with every other day in the same month. We count all of these as misses, and their cost is in each model's total.

Confident answers without checking. Gemini 2.5 Pro answered two questions without running a single query, once with numbers it made up. Claude Opus 4.1 picked one reading of most ambiguous questions and stated it as fact, and made up an industry benchmark when the data didn't have one. GPT-4o's answers read well but often rested on a subtle SQL bug, and on one question its reply trailed off into gibberish.

The newest models still make some of these mistakes, just less often. Today's models also miss data quality problems, and on the hardest hidden patterns most of them stop at the first plausible explanation.

A few caveats

We ran the older models this week, not when they came out, and through OpenRouter's hosts as they are today. For models served by only one host, like Llama 4 Maverick, that host's quality is part of the result. A harness and prompts tuned for older models might get more out of them, but every model on our board gets the same ones.

We tested only a few models from between August 2025 and June 2026, so the flat stretch in the first chart partly reflects what we ran, not just what was released.

One run per model is a single sample. In our repeat runs of other models, scores moved by up to three points between runs, so treat a one- or two-point gap as close to a tie. Our harness has no time limit on a single query, so a query that never finishes counts as a miss rather than an error the model could recover from.

The Test Harness

We run every model in Plotly Studio Embedded, a drop-in analytics agent that runs on Dash apps. This harness is tuned to run data analytics queries and make charts with multi-turn tool calling, with less emphasis on writing text. It's meant to put the data front and center.

Frequently asked questions

How much have LLMs improved at data analysis?

On our exam, the best model in November 2024, GPT-4o, passed 20 of 43 questions. In August 2025, GPT-5 passed 35. Since September 2026, three models have passed all 43: GPT-6 Astra, GPT-6.1 Sol and Claude Fable 5.2.

Is GPT-4o still good for data analysis?

Not compared with what's available now. It passed 20 of 43 questions for $5.83 a run. GPT-6 Luna passes 38 for $0.13, and GPT-6.1 Sol passes all 43 for $2.61.

Is Llama good at data analysis?

Llama 3.3 70B passed 6 questions and Llama 4 Maverick passed 4, mostly because they often wrote their tool calls out as text instead of calling the tools. Meta's newer open-weight Muse Glimmer 30B passes 35 (more in our U.S. open-weight models post).

How much cheaper has data analysis with LLMs become?

GPT-5's score of 35 cost $4.67 a run in August 2025. Today GPT-6 Luna passes 38 for $0.13, about 36 times cheaper.

Which older model did best?

GPT-5 and Claude Opus 4.6 each passed 35, and GPT-5.4 passed 34. Of those, GPT-5.4 was the cheapest at $3.02 a run and the fastest at 13 seconds a question.

About the benchmark

We run new models through the same exam as they're released, and we'll keep sharing what we find. We ran all ten older models through OpenRouter, with the same harness and grading as every other model. We checked every miss against the model's transcript, and changed three verdicts where the answer met the question's pass rule as written.

To see the current models on one chart and in a full results table, along with cost, speed and how the exam works, see Introducing DataAnalyticsBench.

The exam is private and we're sharing it with researchers. We're keeping it private so that the answer key doesn't make its way into the training set. For the same reason, the data is fabricated: the warehouse is a synthetic wind-energy fleet, built to look like a real operator's data, so its answers don't exist anywhere else.

If you'd like access, reach out to Chris Parmer at Plotly.