Day 1 of ML System Design Case Studies Series : ML System Design Basics
Hello peeps! Welcome to Day 1 of ML System Design Case Studies Series. We have successfully finished System Design Case Studies Series.
Pre-requisite to ML System Design Series —
11 Most important concepts of System Design
Complete System Design Case Studies
Complete Data Science and ML with projects Series
Projects Videos —
All the projects, data structures, SQL, algorithms, system design, Data Science and ML , Data Analytics, Data Engineering, , Implemented Data Science and ML projects, Implemented Data Engineering Projects, Implemented Deep Learning Projects, Implemented Machine Learning Ops Projects, Implemented Time Series Analysis and Forecasting Projects, Implemented Applied Machine Learning Projects, Implemented Tensorflow and Keras Projects, Implemented PyTorch Projects, Implemented Scikit Learn Projects, Implemented Big Data Projects, Implemented Cloud Machine Learning Projects, Implemented Neural Networks Projects, Implemented OpenCV Projects,Complete ML Research Papers Summarized, Implemented Data Analytics projects, Implemented Data Visualization Projects, Implemented Data Mining Projects, Implemented Natural Leaning Processing Projects, MLOps and Deep Learning, Applied Machine Learning with Projects Series, PyTorch with Projects Series, Tensorflow and Keras with Projects Series, Scikit Learn Series with Projects, Time Series Analysis and Forecasting with Projects Series, ML System Design Case Studies Series videos will be published on our youtube channel ( just launched).
Subscribe today!
Tech Newsletter —
If you are interested, you can join my newsletter through which I send tech interview tips, techniques, patterns, hacks — Software Development, ML, Data Science, Startups and Technology projects to more than 30K readers. You can subscribe to Tech Brew :
System Design Case Studies — In Depth
Design Instagram
Design Netflix
Design Reddit
Design Amazon
Design Messenger App
Design Twitter
Design URL Shortener
Design Dropbox
Design Youtube
Design API Rate Limiter
Design Web Crawler
Design Amazon Prime Video
Design Facebook’s Newsfeed
Design Yelp
Design Uber
Design Tinder
Design Tiktok
Design Whatsapp
Most Popular System Design Questions
Mega Compilation : Solved System Design Case studies
In this post we will be discussing ML System Design as follows —
ML System Design — basics
Template that you should use for ML System Design
Most popular ML System Design Questions
ML System Design Case Studies in detail
Let’s start with ML System Design Basics ( complete pre-requisites above before jumping to this section)
First things first. What is ML System?
ML System is all about 4 things —
- ML project requirement analysis
- Building Data Pipeline
- Building the model and train
- Serve the model
Let’s talk about each part in detail -
ML project requirement analysis
First understand the problem statement in depth and project that solves that problem. Ask questions!
This consists of 3 steps —
- Establish a Goal — Before starting anything, set a goal. To put it simply, ask — What is the end result?
- Constraints — What are some of the project and performance constraints? Project constraints can be space and deployment timeline, available resources, domain knowledge etc. Performance constraints can be — What performance metrics ( accuracy, false positives etc), speed etc are you taking into consideration?
- Evaluation — What are parameters based on which you want to evaluate your system?
Building Data Pipeline
This consists of 3 steps —
- Data Collection — Do you have sufficient data or is the data available? Is it labeled? The data can structured, semi structured and unstructured as well as can be in different formats.
- Data Cleaning and Preprocessing — It is the process of figuring out incomplete, missing, incorrect and inaccurate records in the data and then fixing/correcting them.
It involves —
Removal of unwanted and invalid information
Fixing issues of unknown missing values
Fixing problems with mislabeled features, classes
Handling the outliers
It ensures —
The data is valid and up to date.
Data doesn’t contain any duplicates or missing values
Data doesn’t contain any numerical outliers
It helps define valid data labels for the data which is categorical in nature.
3. Biases : Data that you have collected can be biased. How do you deal with biases?
Build the model and train
It consists of 2 steps —
- Model Selection — There are many machine learning models that you can choose from. The process of selecting the best machine learning model for your data is called as Model Selection. Model that you have selected should not underfit or overfit your data. We will see Model Selection in detail in the next few posts.
- Train the model — In layman’s terms, once you have selected the best model, by training the model on the data that you have is about extracting the patterns from the data and then evaluating the patterns to predict the results.
- Hyperparameter Tuning — Hyperparameters are those tunable parameters which can directly affect how well a model trains and are set before the learning process begins.
Examples of hyper parameters are —
- Number of Epochs
- Momentum
- Regularization constant
- Learning Rate
- No of branches
- No of clusters
For your ML model, in order to achieve maximal performance, it is very important to understand how to optimize the hyperparameters. The success of a ML project is crucially dependent on the choice of good hyperparameters.
Serve the model in the production
In simple terms, it means hosting the machine learning model either on the on premise or cloud and make their functionalities available via API’s. A model is deployed as a web service which other services can communicate with.
That’s it for now. Day 2 : Coming soon!
In the next part we will see template that you should use for ML System Design.
Read More —
Day 2 : SQL Basics, Query Structure, Built In functions Conditions
Day 4 : Set Theory Operations, Stored Procedures and CASE statements in SQL
Day 6 : Subqueries, Group by, order by and Having clauses in SQL and Analytical Functions
Day 7 : Window Functions, Grouping Sets and Constraints in SQL
Day 8 : BigQuery Basics, SELECT, FROM, WHERE and Date and Extract in BigQuery
Day 9 : Common Expression Table, UNNEST Clause, SQL vs NoSQL Databases
Day 10 : Triggers, Pivot and Cursors in SQL
Day 14 : MySQL in Depth
Day 15 : PostgreSQL inDepth
Anyways, For Day 15 of 15 days of Advanced SQL, we will cover —
PostgreSQL inDepth
Github for Advanced SQL that you can follow —
All the projects, data structures, algorithms, system design, Data Science and ML, Data Engineering, MLOps and Deep Learning videos will be published on our youtube channel ( just launched).
Subscribe today!
System Design Case Studies — In Depth
Complete Data Structures and Algorithm Series
Github —
Complete System Design Series.
6. Networking, How Browsers work, Content Network Delivery ( CDN)
Github —
Subscribe/ Follow, Like/Clap and Stay Tuned!!
Some of the other best Series —
30 days of Data Structures and Algorithms and System Design Simplified
Data Science and Machine Learning Research ( papers) Simplified **
100 days : Your Data Science and Machine Learning Degree Series with projects
Complete Data Visualization and Pre-processing Series with projects
Exceptional Github Repos — Part 1
Exceptional Github Repos — Part 2
Tech Newsletter —
If you are interested, you can join my newsletter through which I send tech interview tips, techniques, patterns, hacks — Software Development, ML, Data Science, Startups and Technology projects to more than 30K readers. You can subscribe to Tech Brew :
For Python Projects —
For complete 60 days of Data Science and ML : Day 1 — Day 60 : Quick Recap of 60 days of Data Science and ML
Follow for more updates. Stay tuned and keep coding!
For other projects, tune to —
Build Machine Learning Pipelines( With Code)
Recurrent Neural Network with Keras
Clustering Geolocation Data in Python using DBSCAN and K-Means
Facial Expression Recognition using Keras
Hyperparameter Tuning with Keras Tuner
Custom Layers in Keras
