What is DataGPT and why should Data Engineers and Scientists care?
Are Data Experts loosing their Jobs due to AI?
The topic is not entirely new but probably will be a trending in the area of Data Engineering and Data Analytics within the next months and years.
Also Microsoft, together with the functions of ChatGPT, have similar approaches with Copilot. Google has also presented a prototype for their data warehouse BigQuery.
Now, DataGPT has introduced what it claims to be the first AI-powered conversational data analyst in the industry. The startup combines generative AI’s creativity and comprehension with advanced analytics techniques’ logic and reasoning to make complex analysis more accessible, particularly for smaller businesses[1][2].
That makes sense because many businesses face challenges in effectively analyzing their data, as current business intelligence tools often lack the iterative querying necessary for in-depth data exploration and uncovering valuable insights. Additionally, consumer-focused generative AI tools like ChatGPT are unable to integrate with large databases, limiting their effectiveness in data analysis[2].
Right now, DataGPT offers connectors to major Data Analytic and Warehouse/Lakehouse tools and services like Google BigQuery, AWS Reedshift, Snowflake & Co.
DataGPT can proactively uncover insights for users across various companies. More non technical users and Data Analysts with limited knowledge of SQL or Python can input natural language questions in a familiar chat interface such as: What is our main target group on the market? The service aims to democratize advanced data analysis, allowing everyday business users to access skilled analysis for the first time.
Like previously said, the approach is not completely new. Google has already offered a similar service for BigQuery in preview already last year and Microsoft is trying something similar with Copilot within Power BI and Azure Data Analytic services. But with DataGPT, now a cloud and source system independent service is available for the market which shows the potential. So the big question here is: Will Data Engineers and Scientists jobs decrease in the future?
Well, at least the approach can ease Data Integration, Transformation and Analytic tasks and enable also normal business users to use powerful services like BigQuery, Redshift, Snowflake & Co. But not all can be automated yet, you still have to integrate the data from various sources to Data Warehouses or Lakes and mostly clean and transform it to business logics, therefore you will still need a Data Engineer. But you also have to admit that within the last months especially the big cloud providers offer more and more easily integration services and follow the approach Zero- ETL which clearly ease these kind of tasks.
The same applies to Data Analysts and Scientists who can also use this kind of tools to perform deeper SQL Analytics or even Machine Learning tasks. But also here, you will still need expertise for developing real precise ML algorithms and also to understand the results and how they are produced. So in my opinion, Data Experts won’t be jobless in the near future but such tools can clearly democratize data integration and analytics and shift some task to the normal business users. In many cases, this can be very beneficial for companies due to the lack of data experts in the market.
Sources and Further Readings
[1] DataGPT (2023)
[2] VentureBeat, DataGPT launches AI analyst to allow ‘any company to talk directly to their data’ (2023)
