Part 1 — Database Technolgy Knowledge
What Skills does a Data Engineer need?
How to increase your Market Value and Salary
To be successful as a Data Engineer and thus increase your market value and salary, you need certain skills. I have thought about illuminating these in more detail in various articles — this time with a focus on database technologies.
The most important task for a Data Engineer is to provide high quality data from source systems to systems like a Data Warehouse, Data Lake or Data Lakehouse for example. This data often comes from systems like an ERP or CRM system, social media or production and is often stored in relational databases. However, also unstructured data from an NoSQL could be possible.
That means that a good Data Engineer is not only good in Structured Query Language (SQL) and topics like relational relationships, modelling and principle of normalization. Here, these articles might be also interesting for you:
In addition, Data engineers also have to be familiar with other types of databases, so-called NoSQL databases which are based on file formats, column-oriented (often used in modern Data Warehouses) or a graph-oriented approach — Read here more about it. Examples for NoSQL databases are MongoDB and Cassandra, hybrid systems often used in Data Warehousing are Amazon Redshift or Google BigQuery.
It is also important to know how to access the systems. Be it in a very primitive export as a CSV or via direct data connections, interfaces such as REST, ODBC or JDBC. Knowledge of socket connections and client-server architectures will pay off. Here, I learned that you should also learn about the characteristics and peculiarities of the source systems. Here, an exchange with product owners or IT staff who have the know-how is always useful. I have provided some useful tips in this article.
But to take one point from it in advance: One of the most important points is to know about unwanted behavior in the source systems. If not taken into account these behaviors could really ruin your data integration processes but also other problems can occur for example duplicate and inconsistent data. Like already said, it’s important to get to know the source systems and their pitfalls.
If you are interested in what a Data Engineer earns for their job tasks, this article might be of interest for you.
I hope that this article has given you a brief overview of an important topic that a data engineer should know about — the knowledge of databanks. It is, so to speak, the cornerstone for a successful data integration. In the upcoming weeks, I will publish more articles with other areas that are essential for a data engineer — stay tuned.
Sources and Further Readings
[1] TechTarget, NoSQL (Not Only SQL database) (2021)