Why I Chose Data Engineering Over Data Science
Despite holding an M.S. in data science, I have found the data engineering path more organizationally impactful, professionally fulfilling and personally lucrative than data science.
In spring of 2021 I graduated with a degree in data science; after a two-month job search I landed my current role as a data engineer in the media industry. Initially I intended to pursue data analyst and data science roles. However, job interviews with my current leadership, support from my team and my personal experiences have led me to embrace the role, necessity and financial potential of the data engineering professional landscape.
Before I outline my reasons for pursuing data engineering over data science it must be noted that, like data science, the scope and responsibilities of data engineer roles vary from organization to organization. Obviously, your experience may be different.
Organizational Impact
There is a misconception, I think, that data engineers only do the janitorial work involved with data preprocessing. This is certainly true. There’s a reason that the structures data engineers build are called pipelines. Data engineer and educator Andreas Kretz even calls his Medium publication Plumbers of Data Science.
While data engineers do help facilitate data science work, there are also many opportunities to work with teams beyond technical departments. Meeting with individuals across an organization to determine pipeline and other data product requirements helps break up the coding monotony and, more importantly, helps give data engineers a sense of how a pipeline, query or other data engineering build impacts professionals across your company and empowers them to do quality work.
Accessibility (There’s No Math)
Perhaps it’s just my personal experience, but from my perspective beginning as a new data engineer is a bit less intimidating than beginning as a new data scientist. If you’ve ever been curious about beginning as a data scientist you might have stumbled across a statistic along the lines of nearly 40% of data scientist candidates have phd degrees. 80% of data scientist candidates have graduate degrees.
Data science requires a working knowledge of statistics, linear algebra, calculus and computer science concepts. This is on top of knowing at least two programming languages, some SQL and cloud technologies.
Since there aren’t many explicit data engineering academic programs at either the undergraduate or graduate levels, there is less of a barrier to entry, academic-wise, for those who want to begin or transition to the field. Also, as a math-adverse data engineer I can say with confidence that the data engineering path involves far less calculations than our data scientist counterparts. Luckily SQL and Python provide functions for basic to intermediate calculations.
Sure, tech stacks spanning multiple cloud applications and programming languages can be nerve-wracking at first. However, technologies like Google Cloud and Amazon Web Services are fairly intuitive. Plus, Google Cloud offers resources and certification paths to help legitimize those DIY data engineers.
Professional Fulfillment
This is more of an existential reason, but that shouldn’t diminish its importance. I don’t believe that your work has to be your life’s passion, but you should at least be fulfilled by what you’re doing. I genuinely enjoy learning new skills and watching my skillset grow as I gain exposure to more of my organization’s technical stack.
If you’re lucky, you may receive feedback (either solicited or unsolicited) that lets you know that your work makes a difference in the scope of your organization. Recently, as a new hire, I was fortunate enough to meet with a c-suite executive who, when I introduced myself as a business intelligence team member, remarked that they rely on our dashboards to stay updated on metrics within the company. When you’re coding for hours or debugging a complex query the work can feel monotonous, but receiving recognition and even praise helps contextualize how data engineering work affects each aspect of the organization.
I’m in it for the Money (Just A Little Bit)
I’d be lying if I wasn’t in tech for financial reasons. With an undergraduate degree in journalism and a few years of internships and hospitality jobs, I determined that I needed to improve myself financially (hence the graduate degree).
Despite the publicity surrounding data science, data engineers make more, on average, than data scientists (data engineers: $137,000 vs. data scientists: $121,000). The average entry-level salary for a first-year data engineer exceeds $85,000. Keep in mind: These are base salaries. Since data engineers and data scientists add significant financial value to an organization it is not uncommon for data professionals to receive bonuses on top of base compensation. Additionally, for those who don’t mind forgoing the security of a full-time role, freelance and contract opportunities abound.
With that said, if you don’t enjoy learning and employing new data engineering techniques and, just as importantly, don’t admire the product and teams you are supporting, it won’t matter how much money you make because you likely won’t be professionally or personally fulfilled.
Flexibility
While a solid, linear career path exists for someone staying in a data engineering position, data engineering knowledge lends itself to several possible career paths.
- Data Analyst
- Data Scientist
- Management
- Academia
If you’re on the fence between data science and data engineering, a good question to ask is: Where do my skills lie? If you are more comfortable with (or more interested in) extracting and preprocessing data from APIs and other applications, you may be better suited to data engineering. I hope my experiences can help demystify a career field that may not be as hyped as data science, but is just as critical.
Create a job-worthy data portfolio. Learn how with my free project guide.
