Is Data Analysis A Dying Career Path?

How Data Analysts Will Evolve Over the Next Five Years

The Fate of the Superstar Data Analyst

If you’ve had the fortune of working with a good data analyst, you’ll know that they’re invaluable. This person is a fountain of knowledge for all your enterprise data needs, from getting access to data to navigating the mishmash of enterprise data sources. They leverage their well-honed SQL and dashboarding skills to artfully generate insights at pace. They understand the nuances of the data sets they work with and can bring order to the data chaos that exists in many businesses. No matter how the data is stored or formatted, or even if the data is apparently non-existent, the good data analyst seems to find a way to deliver you insights that often inspire you. And crucially, they make it look easy.

There is of course another side to this story. Because a good data analyst makes it look easy, they quickly become a victim of their own success with expectations of them becoming ever higher. Most good data analysts I have worked with are inundated with ad hoc requests leading to a never-ending to-do list. They are seemingly forever creating dashboards because every good piece of analysis leads to more questions, which leads to more analysis. And sadly, in my time, I have seen so many superstars burn out.

This is not sustainable and has been tolerated up until now because of a lack of feasible alternatives. However, advances in artificial intelligence have shown that there is another way.

The Birth of the AI Data Analyst

To be frank with you, I believe that the type of superstar data analyst I described is becoming a thing of the past. AI automation of data analysis has arrived, and it will automate away a large chunk of many data analysts’ responsibilities.

We can look to what Google has done with their Data Commons as an indication of what is to come. Right under our noses, Google has already released a natural language interface on top of their open data sets. It’s a tool that essentially gives anybody access to their own personal data analyst, and the closest thing to a production level “AI data analyst” I have seen.

We can ask for univariate analysis:

Or even conduct some bivariate analysis looking at relationships between crime rates and poverty:

You should try it for yourself.

You don’t even have to be Google to implement such tools at small scale. Such is the state of the AI landscape now, anyone with some computer programming experience can quickly build their own prototype AI-analyst using large language models and open-source frameworks such as Langchain.

I demonstrate this in a short YouTube video:

For those interested, I have previously written a technical deep dive on how this can be done.

An AI data analyst has several advantages over the superstar, but the primary (and potentially most potent) is scalability. These analysts will not get burnt out by the volumes of requests they receive, are immediately available, and can work 24/7.

If this is what can be achieved now, imagine the state of the field in the next five years. It seems inevitable then that the role of a data analyst as we know it does not have a long future ahead.

In five years’ time, I predict that the role of data analyst will look much different if it even exists at all. But for data analysts, this is not a bleak outlook for your career prospects, it’s a huge opportunity that you are in a prime position to capitalise on.

Data Analysts will Evolve

Over the next five years, the role of the data analyst will evolve. We have seen that generating insights from data and building dashboards is being increasingly automated by AI removing a large proportion of many data analysts’ responsibilities. Which begs the question, what will fill the gap?

Data Engineering

Let’s take Google as a short case study. If you have used Google’s data commons prior to the natural language interface, you’ll know that it is built on top of a well-managed data infrastructure. It’s this data infrastructure that makes the AI bit possible.

We will see more tools like the ones released by Google surface over the coming years. There will be increasing pressure on enterprises to keep up or get left behind, which opens up opportunities for data analysts to accelerate their careers.

Because many enterprises’ data management and processes are not up to scratch, their data analysts can play a pivotal role in building the appropriate systems. The AI automation bit is like the icing on the cake, once all the data management is in place.

Research from Zippia projects a 21% growth in data engineering roles from 2018–2028.

Decision Science/Data Science

There is a lot of overlap between data science and decision science, that’s why I use the terms interchangeably.

For enterprises that can implement AI assisted data analysis it will become much easier to generate insights. However, a little bit of knowledge can be a dangerous thing. You can easily envisage a scenario in which decision makers who have no formal training on how to interpret insights or even how to ask the right questions have access to insights on demand.

As people who are comfortable with dealing with data, data analysts are in a great position to extend their skills beyond generating insights to decision sciences where they can become experts in interpreting and using these tools to ask the write questions. And with enterprises becoming more data driven, there is a natural pathway here for the most talented to move into executive and leadership roles.

The US Bureau of Labour statistics projects data science employment to grow 35% over the next decade (source).

AI Engineering

With the recent progress we have made with large language models (LLMs), building AI-powered applications is more accessible than ever before. Combining accessibility of LLMs with the open-source frameworks, and growing number of educational resources, there are ample opportunities for data analysts to transition to a career in AI engineering. Data analysts can be crucial to developing the AI-powered data analysts that we have been talking about and even tools that are beyond this.

According to LinkedIn, job postings referencing generative AI technologies have multiplied by 21 times since November 2021 (source, page 3). This isn’t a surprise, and many will dismiss it as a hype bubble. However, I would caution against this type of thinking as even after the initial hype bubble, there is still a plateu of productivity (see the Gartner hype cycle chart below).

Conclusion

It’s time for data analysts to make their move. The role of data analysts may still exist in name, but in five years’ time, it’s likely that the required skills will look very different.

If you’re a data analyst, consider picking up data engineering skills through one of the many bootcamps or online courses available. Awesome Data Engineering has a range of free resources conveniently organised by topic to get you started.

If you’re more inclined towards business, the decision science route perhaps makes more sense. Decision science courses are harder to come by, however there is a LinkedIn learning path on decision intelligence created by Cassie Kozrykov. You can also follow her Substack for free resources on the subject.

For those interested in AI engineering, I will be releasing a course giving you hands on experience developing LLM powered applications with industry best practices. You can register your interest by joining the waiting list for the course.

This is an exciting time to be a data analyst, you are very much at the forefront of the AI innovation and your data skills give you a head start in capitalising on it.

If you’re keen to enhance your skills in artificial intelligence, join the waiting list for my course, where I will guide you through the process of developing large language model powered applications.

If you’re seeking AI-transformation for your business, book a discovery call today.

For more insights on artificial intelligence, data science, and large language models you can subscribe to the YouTube channel.

A Message from AI Mind

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the AI Mind Publication
• 🧠 Improve your AI prompts effortlessly and FREE
• 🧰 Discover Intuitive AI Tools