5 Unsexy Truths About Working in Machine Learning

Things I learned over the last 5–6 years

To help me understand you fill out this survey (anonymous)

I work in Machine Learning. To readers/viewers of my work, this won’t come as a surprise. To people who don’t know me as well, feel free to check out my LinkedIn/articles/videos for a better understanding of my skills/experience. My specialty is in statistical analysis. I’ve had experience working in Road Safety, Health System Analysis, Big Data Analysis for a Bank, disease detection, biometric recreation, and currently work in Supply Chain Analysis.

Recently, I had an IG call with a follower. She was starting out their 2nd year in college and wanted to break into Machine Learning. So we had a 20–30 minute call going over her skills, why she wanted to get into ML (money and job security lmao), how she had planned to proceed, etc. Just to see what she knew about the sector and what she expected it to be like. What struck me was how her expectations about the work were very different to what my experience was.

And when you think about it, such misconceptions make sense. Many people’s first exposure to Machine Learning comes from the following sources: statistics about how big machine learning jobs will be (look above) or sensational news articles (remember the buzz created by AlphaGo? We see some hype in biology machine learning right now). Then they watch a few videos on YouTube about building Deep Learning Models, take a few courses (if any) and that’s it.

Unfortunately, this simplifies the process a lot. And it leaves out a lot of the ugly details. Which makes sense. The courses and videos couldn’t fit all the details that go into the process. Heck, even my articles seldom cover all the relevant details of a paper/process. But thinking that your machine learning experience will resemble the courses is a big mistake. It’s like thinking that the football life of Cristiano Ronaldo comes through in just the games. Sure those are important. But he also has to train, care about nutrition, and do other things that are hidden behind the scenes. And he spends more time on these (he plays 1–2 times a week, but has to train, eat and sleep every day).

In this article, I will share some insights/learnings that should give you a more balanced view of machine learning. Note that while it may come off as negative, I still believe that Machine Learning is a great field to work in/know about. That’s why I started my content (to democratize ML for everyone). I just want to make sure that you consider some of these things that are typically not talked about.

1: You will not make a lot of money

This will leave you scratching your heads for a second. Isn’t Machine Learning all about making a lot of money? And yes starting salaries at a lot of tech firms for Machine Learning People is very high. The statistics back this up. However, this hides an important fact. Most of these roles require at least a Masters degree and generally a Ph.D. This is a snippet of a conversation I had with Lucas, someone working in the industry right now (for a major bank). He’s interacted with a lot of applicants in the field.

He mentions the need for a Ph.D. to be competitive in most major machine learning roles. There are two alternatives he mentions. The latter is transitioning into engineering. Engineering would require more developed coding skills and a greater emphasis on the preprocessing/visualization parts. For more information check out this 60-second video on how to design your projects to get a machine learning/data engineering role. In the video description, I detail a full sample project you could build with modifications.

The former is the route I took. My interest is in statistical analysis and math. So I wanted to work in analysis and evaluation. For most of the last 5 years, I’ve been working contract roles for smaller organizations. By virtue of being smaller, they couldn’t pay me as much. However, it did help me get some money and pay some of the bills. And this eventually lead to bigger and bigger roles. Now that I have gained experience, I attract more lucrative roles. More recruiters have been sending me such messages. I talked to her, and the compensation was more in line with the industry standards.

But this will take time to build. You will spend a lot of the starting working in barely or slightly above minimum wage jobs. Especially if you do online contract jobs where you will work with people from different geographies.

2: You will need to get good at Programming

No, I don’t mean coding fancy pipelines or data transformations. Also don’t mean coding up models using Keras and Tensorflow. I mean the software engineering type of coding. The grind of working with different data structures, figuring out threads, and writing all kinds of functions. I never really studied Python as a language. I learned what I had to for whatever work/project I working on. So imagine my surprise when I took the LinkedIn Skill assessment and got the following result.

The fact is that you will HAVE TO become a good programmer as you get into Machine Learning. Sure autoML tools exist (I have created one myself). But for now, you will most likely be programming everything yourself. You will do a lot of querying, transformations, mapping, sorting, and printing. Data Structures like Graphs, Trees, Multi-Dimensional Lists, and Dictionaries will have to become your best friends. This fact is often obscured in tutorials/courses/articles, where everything is ordered neatly. But irl, you will have a lot of trial and error.

3: You will Grind a lot

Most of your work will be grinding. You will spend an ungodly amount of time just turning your Pandas df into Series into a NumPy array etc. More time will be spent reading through your print statements trying to figure out why your output values aren’t in the expected range. So much effort will be spent on cleaning (print(df.head() appears in my dreams). Trying different network architectures. Machine Learning is way more than models and clever evaluation metrics. It is a full-time job with a lot of grinding.

4: You will take a lot of shortcuts

You know how Google spends millions of iterations in training? How GPT3 was trained with all the internet? Lol, forget that. You won’t have that kind of data and computing power. Your employers will most likely not want the best set of hyperparameters and neural network architecture if it means paying you for 10 extra hours. You’ll find yourself not being completely thorough, taking a lot of shortcuts, or just making exceptions for all kinds of conditions. This does not mean you should submit bad work. Just that you will find yourself submitting work that you know could be better. Most employers prefer a cheap 85% score over a very costly 95%.

5: You will have to go back to the basics a lot

Machine Learning is prominent. It is a subset of artificial intelligence. So it is very interesting to me that ML gains more search than AI. Recently, Deep Learning became more searched than ML, even though DL is a very small part of ML. On the surface, this doesn’t make too much sense. But it does speak to an element of human psychology. People want to get into the new shiny thing, and will often overlook the basics to try getting into it.

When you start working in ML, you will however see why basics are the king. They are generally more generalizable (a general Neural Network has more use-cases than a Recurrent Neural Network). They solve most problems (remember the point about cheap 85%). But you will have to really have to get into the basics. Learn to read learning curves. Figure out cross-validation and multi-metric evaluation. Truly understand how to write reward/cost functions and design incentives. This will help you a lot.

When you start to understand the basics, you can start handling harder problems. An intimate understanding of Neural Networks will let you see when you should add additional bias or apply dropout. You’ll be apply bagging or boosting however you see fit. Being familiar with the basics will help you succeed. My Boxing coach always said you can become a World Champion off your Jab and Movement. Or take Deontay Wilder. He became an elite through his exceptional right hand and power. Basics will be the best investment of your time to do well in a machine learning career.

Closing

Hopefully, this article helped you gain more perspective on what it’s like working on Machine Learning. Again, the point of this article was not to discourage you from getting into the field, but just to give you another perspective on it.

If you liked this article, check out my other content. I post regularly on Medium, YouTube, Twitter, and Substack (all linked below). I focus on Artificial Intelligence, Machine Learning, Technology, and Software Development. If you’re preparing for coding interviews check out: Coding Interviews Made Simple.

For one-time support of my work following are my Venmo and Paypal. Any amount is appreciated and helps a lot:

Venmo: https://account.venmo.com/u/FNU-Devansh

Paypal: paypal.me/ISeeThings

Reach out to me

If that article got you interested in reaching out to me, then this section is for you. You can reach out to me on any of the platforms, or check out any of my other content. If you’d like to discuss tutoring, text me on LinkedIn, IG, or Twitter. If you’d like to support my work, using my free Robinhood referral link. We both get a free stock, and there is no risk to you. So not using it is just losing free money.

Check out my other articles on Medium. : https://rb.gy/zn1aiu

My YouTube: https://rb.gy/88iwdd

Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y

My Instagram: https://rb.gy/gmvuy9

My Twitter: https://twitter.com/Machine01776819

My Substack: https://codinginterviewsmadesimple.substack.com/

Get a free stock on Robinhood: https://join.robinhood.com/fnud75