Why Data Engineering is the Perfect First Job if You Don’t Know Which Data Role to Pursue

How a data engineering job can help you identify strengths, interests and skill gaps.

Don’t Undervalue Your Data Skills

Perhaps the biggest mistake I made as a data science student coming out of graduate school was to limit myself to one kind of job application. As I approached my final semester I realized that while my programming skills were solid, communication skills were honed from journalism and my domain knowledge was applicable to the job postings I targeted, I realized I did not want to jump right into a data science job. The root of my misgivings, I’ll admit, was professional insecurity. Like you, the beginning career data professional, I was intimidated by postings that required me to know four programming languages, supervised and unsupervised learning algorithms and exhibit phd-level math skills.

Even though an individual with a data science degree and applicable skills can still have an impressive and lucrative career trajectory, I couldn’t see myself following that path. At least not for my first job. In that instance, I did what I would highly advise against: I underestimated my professional value and applied to jobs beneath me (that is not meant to sound egotistical. I mean the market value associated with my skills, education level and relevant experience). I applied to and, thankfully, received many interviews from employers looking for a data analyst. Since my favorite part of my data science courses was creating white papers and presenting my findings, I figured this would be a great way to start a data science career. However, I now realize that, contrary to popular belief, a data analyst is not simply a junior data scientist. This position is not necessarily even a stepping stone to a heavier data role, like machine learning or analytics engineer.

Not Quite a Data Scientist

Depending on the industry and company, the data analyst role is much more focused on extracting insights from existing dashboards and datasets. As I participated in data analyst interviews, discussed my work and even presented projects, like some of my data-driven Medium articles, I realized my interviewers were more interested in seeing how I sourced and interpreted data rather than how I applied my more technical skills like Python, R and various machine learning abilities. I reasoned that I would rather have a quote-unquote easy day job so I could focus on filling in some skill gaps from school to up skill and become a data scientist. However, as I progressed in interviews I found out about this new (to me) position called data engineer.

Data Engineers Are Always Learning

If you have aspirations to be a data scientist, data engineering is a career path that closely aligns with and often directly supports the work of a data science team. While business and data analysts learn valuable data analysis and visualization skills, data engineers gain a broader exposure to programming languages, querying syntax, database architecture and even business knowledge. One of the most important pieces of advice I received during a prior media internship is that when it comes to career paths:

It’s just as important to realize what you don’t want to do as it is to find your dream job.

Since data engineers support so many different teams, products and aspects of a business, you will gain wider exposure to opportunities and interesting roles than you would being siloed creating machine learning models or fulfilling ad hoc analysis requests.

The fact that data engineering as a discipline isn’t entirely defined is also helpful. In fact, the role can differ wildly from organization to organization. From completely different tech stacks to the division of responsibility, one organization’s data engineer could be another’s analytics engineer, for example. This isn’t a bad thing. An ever-changing, evolving job title has the advantage that learning and up skilling is pretty much built into the role. I’m constantly learning not only new cloud-based tools, but also new applications of query and programming languages.

Pardon the interruption: For more Python, SQL and cloud computing walkthroughs, follow Pipeline: Your Data Engineering Resource.

To receive my latest writing, you can follow me as well.

Data Engineers Drive Business Decisions

As a data engineer, I also have a better understanding of the logic behind data-driven business decisions. While a data analyst will ask clarifying questions to understand what a stakeholder wants to learn from a dataset and a data scientist will ask what they’re trying to predict, a data engineer gets to know ‘the big picture’ about how the data will advance the goals of the organization. On a more existential level, for me at least, it’s fulfilling to know that my work literally fuels the organization. I mentioned my role to a c-level executive and he remarked that he looks at the dashboards our data populates every morning.

Data engineering is truly the most visible invisible role on the data science spectrum.

Data Engineers Can Do Any Data Role

Since data engineers develop an intimate understanding of how data is collected, cleaned manipulated and presented, individuals moving from a data engineer to a data analyst or data science role have an advantage: They understand what clean, sustainable and effective data looks like and how it is used to make business decisions. Data engineers also create and deploy machine learning pipelines, meaning they work closely with a data science team to help optimize prediction models. Assuming the candidate learns the required machine learning models and brushes up on some statistics, a data engineer could easily transition to either data science or data analysis.

Data Engineering Fosters Collaboration

Data engineering develops your collaboration and problem-solving skills in a way that a data professional simply doesn’t experience in a data analyst or data science role. In addition to creating pipelines, a data engineer also must be responsible and resourceful enough to respond when data is not showing up as expected. Developing not only the problem-solving skills, but also a resolve under pressure, will be invaluable for a data engineering candidate who seeks to progress in the field or transition to an adjacent data role.

Although junior data science roles exist and most anyone with a data science background would make a quality data analyst, I maintain that data engineer is a role that provides opportunities to learn, react under pressure and develop a broader understanding of how data drives business decisions. Given that the industry is currently experiencing a data engineering shortage, I cordially invite you to consider pursuing what is becoming an increasingly vital data role across industries.

Create a job-worthy data portfolio. Learn how with my free project guide.