Yes, Even Data Science Will Be Automated. Here’s How You Can Stay Relevant.

How rapidly advancing AI solutions and emerging low-code platforms could impact the once-bulletproof data workforce.

Hey Alexa/Siri/Google, why did you take my job?

With a worldwide shortage of specialty tech jobs like data engineers and data analysts and rising wages for software developers, machine learning engineers and data professionals across the board, it seems like the tech boom will continue into the roaring 2020s. While I’m fortunate to be among those who have skills that the market considers to be particularly valuable at this moment, I have a lingering fear that the good times won’t last forever. However, after some panic-Googling I managed to determine that the general sentiment behind the clickbait headlines is that yes, a percentage of IT jobs will be automated, but there will still be a role for skilled tech professionals — it just might not be found under a mountain of code.

First, let me clarify. Even though I am apprehensive about how the coming automation wave will impact the data landscape, I am a proponent for automation. Being able to find an automated solution for a tedious, repetitive task is incredibly satisfying and impactful on an organization. One of my favorite projects I‘ve worked on as a data engineer was the automation of how I inventory and clean a database. We managed to cut manual labor hours from 5–7 days down to the few minutes it takes the script to run and for there to be a manual review of its output. Not having to manually click through and delete tables saves both time and sanity.

However, the above anecdote exposes a rarely spoken truth: Tech jobs, even so-called ‘hot’ jobs like data science and data engineering, can, in fact, be automated. Look no further than the case of an anonymous software engineer who revealed that he automated his job for five years. In doing so, this programmer worked, by his estimate, 50 hours in the course of a year when his colleagues would put in 50 hours per week. For years, the idea of automation in tech was confined to the individual level. A software engineer might create a Python script to send an email. A data engineer might set up a pipeline that can be automated using a triggering mechanism like a Google Cloud Function.

The Evolution of Automated Data Solutions

So many advancements that we, the technical workforce, still perceive as novel are actually fairly outdated by the industry’s breakneck pace of innovation. Cloud computing, cloud architecture and cloud engineering are all the rage these days. However, would it surprise you to learn that cloud computing is nearly 20 years old? The term data engineer, now a ‘hot’ job title, is over ten years old, first gaining wide usage in 2011. Aside from these isolated incidents, we’ve seen the advancement of job roles and corresponding specificity of titles. For instance, cloud engineers have replaced database architects and machine learning engineers are joining the ranks of data scientists.

It’s important to note, too, that each iteration of the previously described job is edging toward the precipice of automation. Data engineers were once tasked with establishing and configuring actual on-premise databases (back when they were called database architects or database administrators). Now they only manage the top layer of cloud computing infrastructure. Data scientists used to have to build custom models for every use case; now data science teams can rely on packages like TensorFlow for deep learning needs. Products like Tableau, Looker, PowerBI and Google DataStudio have empowered ordinary business users and data consumers to create professional-grade visualizations with little to no code. The push for no-code solutions has also permeated data science, pioneered by companies like SparkBeyond which seeks to optimize not only the data collection process, but also the feature selection portion of the data science workflow.

Even jobs that you would think would require some degree of code are striving for code-less workflows and platforms. In the data engineering space there are several no-code ETL platforms like Adverity and Hevodata that purportedly do the same heavy lifting of API connection and pipeline construction as a data engineering team. Incidentally, both software engineering and data collection were listed on a Forbes (full disclosure: Forbes is my employer) list of jobs that might be automated in the next decade. Even if we don’t reach total automation, it’s possible that the abundance of coding educational material will empower more individuals to learn how to code, implicitly decreasing the value of a skillset that was once considered to be a complex skill to acquire and master.

Embrace Low-Code and No-Code Movements

Proponents of this low-code or no-code movement don’t envision a loss of jobs in the same way other sectors like manufacturing or customer service may experience. Instead, they choose to envision a more utopian future in which low-code or no-code platforms free data professionals up to do more interesting work and organizationally impactful work. There’s a reason Al Sweigart entitled his book ‘Automate the Boring Stuff with Python.’ A no-code solution to preprocess data or select features could provide a data scientist with more time to think about how to solve business problems. The simplification of code means that the problems data professionals focus on can and will become increasingly complex. In order to thrive in an era of automation, it will become necessary to embrace or at least strongly consider the relevance of these no-code initiatives.

Become an Expert in AI Ethics

Looking further into the future, data driven roles might evolve beyond coding and problem solving entirely. Even with less code and less business problems, there will still be intriguing and vexing questions to answer regarding data privacy and the ethics of developing and deploying AI solutions. For what it’s worth, you could take billionaire Mark Cuban’s advice and, while your job becomes automated, spend your time earning a philosophy degree. Acquiring an in-depth knowledge of philosophy could mean a lot more in the future than it does for liberal arts graduates today. According to Mark Cuban, liberal arts degrees could be worth more than computer science degrees in the near future. Reputable universities like Cambridge are offering degree programs specifically for those interested in addressing the Ethics of AI. While data scientists soon may not be writing the code that powers these AI solutions, they will need to understand and determine appropriate use cases for an increasingly powerful and controversial technology.

Understand Data Privacy

With an increasing number of complex privacy laws, such as the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA), there will also be a need for individuals that understand both the data science field and the associated legal framework. Being able to toggle analytical cookies on nearly every website will be the norm, but there has to be an individual or team at each company that can determine what information is acceptable, legally and ethically, to collect. Additionally, data professionals will start to find (if they haven’t already) that while overall coding might decrease, data privacy and overall infrastructural security will become a more significant aspect of their day-to-day job responsibilities. In the past, organizational stakeholders requested every last bit of data on users. However, with increased scrutiny surrounding data collection, it’s possible that a significant portion of a data analyst or data engineer’s job will be consulting with internal stakeholders about how they can and can’t use customer data to achieve business objectives.

Become More Than a Coder

While the ability to code isn’t as foundational to a data scientist’s identity as it is to a software engineer, considering the thought that once-complex processes, like developing and deploying ML models, could be streamlined and automated is an unsettling thought. However, this is also an opportunity to realize, as cliche as it sounds, that data professionals are more than coders. There is a wealth of domain knowledge and STEM knowledge required to be a data analyst, scientist or engineer. Unlike software engineers, who use code to build, data professionals use code to derive insights.

If data professionals are still able to derive these insights through code-less platforms there will still be a need for individuals who understand the logic behind models and analytic techniques. As the data industry gradually adopts more no-code solutions, there is an exciting opportunity for data professionals to take on more of a leadership role within their respective organizations. If data engineers don’t have to spend so much time writing code, they can dedicate time to explaining data use cases to stakeholders and take ownership of their datasets to ensure ethical and legal compliance.

When you have to write multiple ETL, EL, or ELT pipelines during a sprint, there is simply not enough time to have that kind of granular focus on the business application. This will open up more opportunities for data professionals to have more visibility and clout within their organizations. This is a good thing. In an automated future, data may no longer be a so-called back-of-shop operation.

Data professionals should not fear automation, but they should know that it’s coming.

Create a job-worthy data portfolio. Learn how with my free project guide.