Building a Production-Ready Text-to-SQL System: Achieving 88.15% on Spider 1.0

Building a production-ready Text-to-SQL system requires much more than generating SQL with a large language model. Enterprise applications demand accurate schema understanding, context retrieval, validation, and error correction to produce reliable results on previously unseen databases.

In this study, we benchmark our Data Agent on Spider 1.0 and achieve 88.15% Execution Accuracy using the Gemma 4 26B model. Rather than focusing solely on benchmark scores, this article shares the architectural decisions, engineering lessons, and benchmarking insights behind these results.

Large language models have significantly improved SQL generation, but building a production-ready Text-to-SQL system requires much more than translating natural language into SQL. Enterprise applications must understand complex database schemas, retrieve the right context, validate generated queries, and produce reliable results on databases they have never seen before.

To evaluate these capabilities, we benchmarked our Data Agent on Spider 1.0, the most widely used Text-to-SQL benchmark. Using the Gemma 4 26B model, our system achieved 88.15% Execution Accuracy, demonstrating that architecture and context management are just as important as model selection.

Choosing the right benchmark is just as important as choosing the right model. While newer datasets such as Spider 2.0 and BIRD introduce more complex enterprise scenarios, Spider 1.0 remains the most widely adopted benchmark for evaluating Text-to-SQL systems and comparing results across the literature.

For our initial evaluation, we selected Spider 1.0 because it provides a standardized and well-established baseline. It allows us to measure how effectively our Data Agent understands previously unseen database schemas and generates correct SQL under realistic conditions.

Figure 2 summarizes how Spider 1.0 compares with other commonly used Text-to-SQL benchmarks.

Achieving high Text-to-SQL accuracy is not simply a matter of using a larger language model. Throughout our benchmarking process, we found that architecture and context management had a greater impact on performance than model selection alone.

Our Data Agent is designed as a multi-stage pipeline that progressively narrows the problem before asking the LLM to generate SQL. Instead of exposing the entire database schema to the model, the system first retrieves only the relevant schema and contextual information, then generates a SQL query, validates the result, and automatically corrects common errors before execution.

As illustrated in Figure 3, the architecture consists of three main stages:

• Information Retrieval – The user’s question is analyzed to identify the relevant database objects, and only the necessary schema information is retrieved from the metadata store.
• Self-Refining Agent – The language model generates SQL, validates the query against the retrieved schema, and automatically refines it whenever validation fails.
• Execution & Response – Once validated, the SQL query is executed against the target database, and the results are returned as structured data or a natural-language response.

This production-oriented architecture is inspired by ideas introduced in DIN-SQL and DAIL-SQL, while being adapted for enterprise environments where reliability, explainability, and robustness are as important as benchmark accuracy.

Selecting the right language model was an important part of our benchmarking process. Rather than evaluating a single model, we experimented with a wide range of open-source LLMs, including Llama 3.1, Qwen 2.5, DeepSeek, SQLCoder, CodeGemma, Mistral, GPT-OSS, and Gemma 4.

To ensure a fair comparison, we evaluated every model using Execution Accuracy (EA), the standard metric for Text-to-SQL benchmarking. Unlike syntax-based metrics, Execution Accuracy measures whether the generated SQL returns the same result as the reference query, making it a better indicator of real-world performance.

Our benchmarking journey is illustrated in Figure 4. We began by experimenting on a local workstation equipped with an RTX 4060 (8 GB VRAM), where smaller models allowed us to iterate quickly and validate our pipeline. After establishing a stable baseline using a subset of 233 Spider 1.0 questions, we moved to cloud GPUs to evaluate larger models on the complete 2,147-question benchmark. This staged approach enabled us to optimize the architecture efficiently before investing in large-scale benchmark runs.

We evaluated the final models on the complete Spider 1.0 test split containing 2,147 questions. As shown in Table 1, Gemma 4 26B achieved the highest score (88.15%), followed by GPT-OSS 20B (86.63%). These results demonstrate that modern open-source LLMs, when combined with an effective retrieval and validation pipeline, can achieve highly competitive Text-to-SQL performance. 

Benchmark scores are meaningful only when they can be compared with previously published work. To better understand the significance of our results, we reviewed both recent Text-to-SQL studies and widely referenced benchmark papers.

Table 2: The accuracy rates of closed-source models reported in the SQL-of-Thought  study on a subset of the Spider dataset; it shows, in comparative terms, that the 88.15% result obtained in this work is positioned above GPT-4o Mini (87%) and at a level close to GPT-5 (89%).

Recent work such as SQL-of-Thought [1] reports that leading closed-source models—including Claude Opus 3, GPT-5, and GPT-4o Mini—achieve high accuracy on Spider benchmark subsets. While direct comparisons should be interpreted carefully due to differences in evaluation settings, prompts, and benchmark subsets, our 88.15% Execution Accuracy demonstrates that open-source models can achieve performance within the same competitive range.

Although these results come from different evaluation settings and should not be compared directly, they provide useful context for understanding the current performance landscape of Text-to-SQL systems. Our benchmark demonstrates that open-source models can now compete with leading closed-source alternatives when combined with a well-designed architecture.

We also compared our results with production-oriented Text-to-SQL frameworks such as DIN-SQL[3] and DAIL-SQL, which combine large language models with techniques including schema linking, prompt engineering, self-consistency, and error correction.

Table 3: The Execution Accuracy results of different methods on Spider 1.0 in the "Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation" [2] study; it reveals the impact on accuracy not only of model selection but also of techniques such as prompt engineering, schema linking, and self-consistency.

Our findings reinforce a conclusion shared across the recent literature: high Text-to-SQL performance depends not only on the language model, but also on the surrounding architecture. Effective schema retrieval, context management, validation, and self-correction are often as important as the model itself.

Benchmarking is not only about measuring accuracy—it is also about understanding where and why a system fails. Throughout our evaluation, we analyzed incorrect predictions and identified several recurring patterns that affected the final scores.

Some errors originated from the benchmark itself. In a small number of cases, inconsistencies in the reference (gold) SQL queries caused logically correct predictions to be marked as incorrect. Other cases involved ambiguous natural-language questions or tie-breaking scenarios, where multiple SQL queries could legitimately produce equally valid answers.

We also observed several model-specific behaviors, including unnecessary type conversions, inconsistent handling of case sensitivity, and occasional overcomplication of otherwise simple SQL queries. These findings reinforced the importance of incorporating validation and self-correction mechanisms into the Data Agent pipeline.

Overall, our analysis showed that many remaining errors were not caused by limitations of the language model itself, but by ambiguities in the benchmark or edge cases in SQL generation. This suggests that further improvements in retrieval, validation, and context management could push the system beyond the 90% Execution Accuracy threshold.

High accuracy is only one aspect of a production-ready Text-to-SQL system. In enterprise environments, infrastructure cost, data privacy, and deployment flexibility are equally important considerations.

One of the key advantages of our approach is that it relies entirely on open-source language models. Throughout the development process, we were able to perform most experiments locally using a consumer-grade RTX 4060 (8 GB VRAM), allowing us to iterate rapidly without incurring any API or token costs. Larger benchmark runs were then executed on dedicated cloud GPUs only after the architecture had been validated.

We used RunPod Secure Cloud for the full Spider 1.0 evaluation. The Gemma 4 26B benchmark was executed on an A100 PCIe 80 GB GPU, while GPT-OSS 20B ran on an RTX 6000 48 GB GPU.

The infrastructure costs are summarized in Table 4.

Although the largest benchmark runs required cloud infrastructure, the overall development process remained cost-effective thanks to local experimentation and open-source models. More importantly, the resulting architecture can be deployed entirely within an organization’s own infrastructure, eliminating dependency on external APIs while providing greater control over data privacy, compliance, and operational costs.

This makes the architecture particularly well-suited for organizations operating in privacy-sensitive, regulated, or air-gapped environments, where data security and deployment flexibility are critical requirements.

Benchmarks provide an objective way to evaluate AI systems, but they are not the ultimate goal. A high benchmark score is valuable only if it translates into reliable performance in real-world applications.

Throughout this study, we found that building a production-ready Text-to-SQL system is about far more than generating SQL. Understanding database schemas, retrieving the right context, validating generated queries, and refining incorrect outputs proved to be just as important as the language model itself.

Our 88.15% Execution Accuracy on Spider 1.0 demonstrates that open-source models, combined with a well-designed architecture, can deliver highly competitive performance for enterprise Text-to-SQL tasks. More importantly, it validates the design principles behind our Data Agent, which was built to operate reliably on complex enterprise databases rather than only on benchmark datasets.

Spider 1.0 represents an important milestone, but it is only the beginning of our journey. Our next steps include evaluating the Data Agent on Spider 2.0, larger enterprise-scale benchmarks, and, most importantly, real customer environments where database schemas are significantly larger and business questions are far more complex.

Our long-term vision is not simply to build a better SQL generator. We aim to build intelligent Data Agents that enable organizations to interact with enterprise data as naturally as they communicate with a colleague—combining retrieval, reasoning, validation, and explainable AI into a reliable decision-support platform.

Benchmarking is not only about measuring accuracy—it is about understanding why a system succeeds or fails.

Throughout our evaluation, we identified several recurring error patterns. Some originated from ambiguities in the benchmark itself, including inconsistencies in reference SQL queries and underspecified ordering conditions. Others reflected common LLM behaviors, such as unnecessary type casting or inconsistent handling of case-sensitive values.

These observations reinforced one of the central findings of our work: architecture matters as much as model selection. Retrieval, schema linking, validation, and self-correction contributed just as much to the final performance as the underlying language model.

While benchmark limitations inevitably affect the final score, our analysis suggests that further improvements in context management and validation could push the system beyond the 90% Execution Accuracy mark.

Spider 1.0 provided an important baseline for evaluating our Data Agent, but it represents only the first step toward production-ready enterprise AI.

Our next focus is to evaluate the architecture on more challenging benchmarks such as Spider 2.0 and, more importantly, on real enterprise databases where schemas are significantly larger, documentation is often incomplete, and business questions are far more complex.

Our goal is not simply to build another Text-to-SQL system. We are building intelligent Data Agents that combine retrieval, reasoning, validation, and explainability to help organizations interact with enterprise data naturally and reliably.

References

[1] Saumya Chaturvedi, Aman Chadha, Laurent Bindschaedler arXiv. “SQL-of-Thought: Multi-agentic Text-to-SQL with Guided Error Correction.” arXiv. June 2026 https://arxiv.org/abs/2509.00581

[2] Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, Jingren Zhou  “Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation” arXiv. June 2026

https://arxiv.org/pdf/2308.15363 

[3] Mohammadreza Pourreza, Davood Rafiei. “DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction” arXiv. June 2026

https://arxiv.org/pdf/2304.11015

Dilşen YILDAR HAVAYLAR, 

Sr. Backend Engineer