Better than GPT-4 for SQL queries: NSQL (Fully OpenSource)
NSQL is a new family of open-source large foundation models (FMs) designed specifically for SQL generation tasks
Raise your hand if you have tried to use ChatGPT or any of the other LLMs to generate SQL queries. I have, and I currently am! but I am thrilled to share with you that a new family of open-source large foundation models (FMs) designed specifically for SQL generation tasks has been released. Its name is NSQL. It has got multiple versions starting from NSQL-350M, NSQL-2B, and NSQL-6B. NSQL-6B outperforms all existing open-source models on Standard SQL benchmarks and even commercial general models like ChatGPT, GPT3, GPT3.5, and GPT4.
Off the Shelf vs Fine tune?
Why SQL; SQL is still the most commonly used language. You may be using different flavors of SQL, from Oracle, MySQL, PostgreSQL, to MSSQL; but SQL is still universally used. Wouldn’t it be great if we could write SQL queries by asking a large language model? It will offset so much of work and probably democratize access to insights to almost everyone in the company who needs it.
How can we make that happen? Usually, we think about either of the two, an off-the-shelf model or fine-tuned Foundation model for a specific task. Both have pros and cons, like off-the-shelf models are trained on the public data and lack the knowledge that is specific to your organization.
If you are like me, you know what I mean. Every company, business unit, and even sometimes engineer (unfortunately) can have their own naming convention for various databases, schemas, table names, and fields. Knowing what each field means is 80% of the battle that we all struggle through in the first 3–6 months of the training process that starts will onboarding and apparently never ends. But all jokes aside, it is truly the issue with the open-source models developed using public data. Customizing these models using organization-specific data is the for the data scientists and the opportunity for the business. This allows the model to be more flexible with the data that each organization has to offer.
One could use a mega model like GPTs and think that it will work if I just provide the context. Well, I say you are not 100% wrong, but still, you will be surprised how many times the model hallucinates (even at temperature=0 settings). Also, asking a general model of the size of 500B parameters to write a specific SQL query might be a job not tailored to the resources used. Then what is a solution?
A Fully OpenSource FM specifically for SQL tasks
AI is just picking up speed, and the adaption of the AI has been fairly slow in companies. Some industries are primed for its adaption, but there is a lot of lag in terms of expertise, infrastructure, and in general, willingness to invest in AI-based data solutions. As a matter of fact, very few enterprises have the ability to build their own code FM. This means that they will need an open or closed-source alternative to fill the gap. There are a few specific gaps when it comes to using such off-the-open/closed-shelf models.
They lack personalization, as discussed before. Each data at each company and its codebase is almost always unique. That means that the one model that fits all might not be the best idea.
Additionally, there are many programming languages in use at enterprises in various departments. The models developed to handle many languages can work as a generalist but not as a specialist. Sometimes that means that you “almost” got the answer, but the “almost” made it completely and utterly unusable.
Privacy is one of the big issues too. Companies are often hesitant to use a 3rd party engine to drive the AI-bus. They would rather not share anything with the company that provides then them with the AI model. This means that it is challenging to actually build things that are specialized in the task. If a poll is taken, I bet most enterprises today would rather use their own hardware running the show than a 3rd party (especially if the tasks are privacy-focused).
NSQL Approach to Answer Your Questions
NSQL has executed the following solution. The first step was to collect a lot of data. This is the corpus of general SQL codes from the web using self-supervised learning. The input data to the model training process contains both the SQL code as well as the data schema using labeled text-to-SQL pairs. This was further trained to follow the commonest specific tasks and provided with the codes that execute these tasks. This is the fine-tuning process for a general SQL model to be more attuned to the commonest asks that a person may ask the database.
NSQL: Data and training details
To train NSQL 2 datasets were created.
- A pre-training dataset composed of the general SQL queries (The data is available on huggingface here https://huggingface.co/datasets/bigcode/the-stack)
- A fine-tuning dataset composed of text-to-SQL pairs (These labeled text-to-SQL pairs came from 20 different public sources across the web and ranged from WiliSQL to Medical data like MIMIC_III (read more about MIMIC series here https://readmedium.com/what-is-mimic-iv-newer-better-modular-release-of-mimic-4083e3cb14fe)
Here is the full list of fine-tuning databse
This data was augmented with schema augmentation, SQL cleaning, and the generation of better instructions. A total of 300,000 samples were included in the fine-tuning data. The data will be available on HuggingFace.
The training was performed as a 2-step process with pre-training using just SQL codes as the first step and then fine-tuning using the text-to-SQL pairs with schema.
Performance
NSQL was evaluated on two standard text-to-SQL benchmarks:
The Spider
The Spider benchmark contains questions from over 200 diverse databases ranging from actors to vehicles. Read more about it here https://yale-lily.github.io/spider.
Geoquery
The GeoQuery focuses on questions about US geography. Read more about GeoQuery here
Verdict
NSQL version 6B performs better than all existing open-source models by up to 6 points on spider Execution Accuracy benchmarks.
Compared to the closed-source models, the 6B version shows even better results with a healthy lead of 9 points on the spider execution accuracy benchmark (compared to ChatGPT) too. Compared to the Salesforce CodeGen model, both pre-trained (not-fine-tuned) NSQL performs better as by 35-60% while the NSQL instruct (fine-tuned) model performs 100 to 400% better.
Qualitative analysis:
The researchers also conducted the model evaluation using qualitative methods. The result of such evaluation was that the NSQL model could follow instructions better, and additionally, it hallucinates less than any existing model. This is especially true when the schema is provided as the context. The NSQL model functions better if the instructions are explicit. Here is an example of cases where the model performed well and some (5,6) where the model failed pretty badly given that the revenue in context was not defined.
Schema creation code —
Qualitative evaluation.
What does this mean?
A few days back someone reached out to me that “Hey I work in healthcare and we can't send the data out to any other party. We would like to generate queries based on the user input. I was sure that I could do it using the off-the-shelf models but now I am confident that I will be able to do it.
Maybe the performance of this pre-trained model will be less than what I expect, but I could fine-tune it using company data or I could simply provide employee training to allow them to use the tool the right way.
All credits go to NnumbersStation.ai for bringing the power of NSQL to the people. This release is completely free for both academic and commercial work, which makes it an amazingly flexible model to start playing with. Also, don't forget the power of people. When LLaMa’s weights got leaked we got every animal model from the Kindom Mammalia! I think now we should expect a ton of derivative models from this release of NSQL.
Where can I play with the model?
The NSQL-350M, NSQL-2B, and NSQL-6B are all available on HuggingFace, including model weights. You can use the playground with the 350M model (the rest of the models are too large to be loaded into the inference API)
Where can I find the starter code?
On GitHub, you can find notebooks that walk through the setup and work for Postgres and Sqlite SQL flavors. The repository and the examples are not well set up. So expect hiccups like the manifest package was noted to have version 0.1.9 in the requirements, but only available versions are up to 0.1.8. Make adjustments before you run it locally using SQLite (I recommend SQLite as it will not require you to create a Postgres database and user just to get started.
Good luck! And Have fun with it.
If you have read it until this point — Thank you! You are a hero! Sorry for the short and possibly hasty article this time. I try to keep my readers up to date with “interesting happenings in the AI world,” so please 🔔 clap | follow | Subscribe
Become a member using my referral: https://ithinkbot.com/membership
Find me on Linkedin https://www.linkedin.com/in/mandarkarhade/
