Machine Learning Art

How to reduce costs of Machine Learning

3 easy to use tools for cost reduction

What would happen if one could make training NLP models 10 times more efficient without any consequences? More importantly, can we have simple instructions for doing so? What do the collective cost, time, and energy savings be in addition? There is no one solution to this. We need a lot of different innovations. I will mention some toolkits that make this possible today

🔵 Originally published at MLearning.ai Art in March 2022.

I’m guessing that many NLP — deep learning models are trained every day. The cost, energy, CO2, and time for training deep models are not trivial. Components of modern graphics processing units (GPUs) are counted, energy-hogging memory accesses are discussed, and consequences for deep learning applications are identified.

Just how much does it cost to train a model?¹

• $2.5k — $50k (110 million parameter model)
• $10k — $200k (340 million parameter model)
• $80k — $1.6m (1.5 billion parameter model)

I will list a few (3) toolkits (reductions time and costs) that are easy to use now.

1. Fast Forward Computer Vision²: FFSV is a drop-in data loading system that dramatically increases data throughput. Train models at a fraction of the cost with accelerated data loading! ffcv is a drop-in data loading system that dramatically increases data throughput in model training:

• Train an ImageNet model on one GPU in 35 minutes (98¢/model on AWS)
• Train a CIFAR-10 model on one GPU in 36 seconds (2¢/model on AWS)
• Train a $YOUR_DATASET model $REALLY_FAST (for $WAY_LESS)

Keep your algorithm the same and just replace the data loader. Peruse these improvements:

🟠 https://github.com/libffcv/ffcv

Installation

conda create -y -n ffcv python=3.9 cupy pkg-config compilers libjpeg-turbo opencv pytorch torchvision cudatoolkit=11.3 numba -c pytorch -c conda-forge
conda activate ffcv
pip install ffcv

2. A PyTorch Library for Efficient Neural Network Training³ Composer is a library written in PyTorch that enables you to train neural networks faster, at lower cost, and to higher accuracy. Composer connects 20+ methods for racing up DNN training for Computer Vision tasks. The state-of-the-art in efficient neural network training.

Composer features:

• An easy-to-use trainer that has been written to be as performant as possible and integrates best practices for efficient training.
• Functional forms of all of our speedup methods that allow you to integrate them into your existing training loop.
• Strong, reproducible baselines to get you started as quickly as possible.

🟠 https://github.com/mosaicml/composer

Installation

Composer is available with Pip:

pip install mosaicml

Alternatively, install Composer with Conda:

conda install -c mosaicml mosaicml

3. COResets and Data Subset selection⁴ CORDS is built on top of PyTorch. CORDS uses recent advances in data subset selection, particularly ideas of coresets and submodularity select such subsets.

The features

• 3x to 5x speedups, cost reduction, and energy reductions in the training of deep models in supervised learning
• 3x+ speedups, cost/energy reduction for deep model training in semi-supervised learning
• 3x to 30x speedups and cost/energy reduction for Hyper-parameter tuning using subset selection with SOTA schedulers (Hyperband and ASHA) and algorithms (TPE, Random)

SpeedUps in Supervised Learning

🟠 https://github.com/decile-team/cords

Installation

To install the latest version of the CORDS package using PyPI:

pip install cords

From Git Repository

To install using the source:

git clone https://github.com/decile-team/cords.git
cd cords
pip install -r requirements/requirements.txt

In conclusion, I think this is an inspiring direction! However, I believe that in achieving 10x+ speedup, multiple approaches need to be combined, and algorithmic innovations should complement solid engineering! By far, the most costly tasks today are Hyperparam and AutoML. I may have missed many other libraries, and I sincerely apologize! However, if you know of libraries with algorithmic ideas for computationally efficient NLP training, please count them in the comments!

I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on 🔵 MLearning.ai 🟠

• Check out my instagram with new material every week
• If you enjoyed this, follow me on Medium for more
• Want to collaborate? Let’s connect on LinkedIn
• https://linktr.ee/datasculptor
• 3D Machine Learning generated model on sketchfab

Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.

All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai

1. Linkedin (6.9K+ ML-professionals)
2. Twitter (4.7K+ followers)
3. Instagram (2.2K + followers )
4. Sketchfab * — individual vRooML!
5. Facebook
6. Youtube
7. Apple Podcasts
8. Substack

🔵 Submission Suggestions

[1]https://arxiv.org/pdf/2004.08900.pdf THE COST OF TRAINING NLP MODELS

[2]Github: https://github.com/libffcv/ffcv

[3]https://github.com/mosaicml/composer

[4]https://github.com/decile-team/cords

🔵 Originally published at MLearning.ai Art in March 2022.