Types of Data Engineering Projects You Should Be Aware Of (to progress in your career)
Avoid repeatedly doing the same kind of project and progress in your Data Engineering career.
After about a year into my Data Engineer position, reading a tonne of articles and people sharing their experiences, I have come to realise that there are different types of data engineering projects.
If you are interested in continuously developing your skills as a Data Engineer, you should try to expose yourself to different projects to:
- expand your knowledge and experience,
- increase your employability and,
- get higher pay.
After reading The Seattle Data Guy’s post, I reflected upon the types of projects that I have worked on and currently working on.
In this article, I will
- expand on the 3 main types of data projects that he outlined,
- other types of data projects
- how to get exposure to different types of projects
The key projects are:
Analytics Project
Most data engineering projects are ultimately about finding insights from the data. The types of pipelines that you are building are potentially for:
- Descriptive/Diagnostics/Analytics
- Predictive/Prescriptive Analytics
If you are in a small company or a company that is still in the early data maturity stage, the chances are you will be doing mainly Descriptive/Diagnostics Analytics. If you are a junior data engineer, you will still get valuable experience from building simple ETL/ELT pipelines, Data Warehouse, and Data Lake for downstream analytics and tools required for these types of projects.
Here’s a good illustration of a data maturity model of an organisation from Gartner:
Automation Projects
This can be a range of things like web scraping, data wrangling, data extraction from pdf, images, audio, and so on.
Traditionally, companies would have their employees spend hours/days/months listening to audio files, watching videos, looking at pdfs, images, excel files, etc to record data manually.
Imagine you can save the company so much time and money by automating this process.
Product Development
A simple data product can be developed using PowerBI. Companies of the same industry could be using the same software or have the same business model. Say, if you developed a dashboard for 1 company, you can package it up and on-sell it to other companies of the same kind.
A more complex data product can be a tool to compare prices from various sources, provide recommendations, predict customer behavior, and so on. The data pipelines for these can be more complicated to build and maintain. This depends if your company would have the budget for R&D to develop this kind of product specific to the industry.
Other Types of Projects
In addition to what The Seattle Data Guy mentioned above, I have broken them down further.
The below will distinguish you between a Big Data Engineer vs Data Engineer.
Cloud vs On-Premise vs Hybrid
Although cloud computing is a big thing nowadays some companies are still in transitioning phase. So having on-prem knowledge is also important.
Moreover, some companies with critical infrastructure (such as the military, water provider, energy provider etc) might have their own data centers and will never move to a public cloud like Azure or AWS.
Big Data vs Small Data
Most projects are small data. If you want exposure to Big Data projects, it requires a different set of tools, skills, and knowledge such as parallel processing, different types of table joins, table types, optimization methods, and so on.
Structured vs Semi-structured vs Unstructured Data
Have you only been handling traditional structured data do you want to handle geospatial data, audio files, video files, etc? Different types of data will give you exposure to different storage solutions and tools you need to know to handle them.
Relational Databases vs NoSQL Databases
Data Engineers need to know SQL to query data from both relational and NoSQL databases. Most companies have relational databases. NoSQL databases can be a document, key-value stores, column-oriented, and graph where you need to know how to store, query, and transform them to a consumable state.
Stream Processing vs Batch Processing
Most projects are for batch processing say, hourly, daily, weekly, monthly, yearly, and so on. Knowing how to handle stream data requires different knowledge and tools as the data is in motion and it needs action immediately when an alert is triggered e.g. noise monitoring, gaming streams, credit card fraud detection etc
How to work on the different types of projects?
In my opinion, it depends on:
- the company you are with (big or small),
- the data maturity level of the company,
- the sector you are in, and
- how aggressive you want to pursue the opportunity.
It is important to be aware of the different types of projects out there so that you can focus on the ones you are interested in and formulate a plan to achieve them.
If you find yourself repeatedly working on the same type of project such as descriptive analytics handling small and structured data for batch processing (which is a good experience for a start), you should find ways to get different exposure if you want to continue to progress to be a Big Data Engineer.
Some companies may not have reached the data maturity level and/or have the budget to give you the kind of projects you want.
Big data and cloud infrastructure are expensive and small companies might not have deep pockets to build and maintain them.
For me, I got the opportunity to work with different kinds of projects i.e. NoSQL databases, Big Data, unstructured data, stream processing, and new product development projects when I work for a multi-national company.
Finally…
If you have other types of projects and experiences, please don’t hesitate to leave a comment.
I hope you find this article insightful and have given you a clearer picture of what you want to achieve in your data engineering career.
Personally, I am still on a journey and am interested to hear from YOU, your journey!
Please follow me on Medium as I continue to share with you my experience!
Your support is appreciated!