What is Neural Architecture Search? And Why Should You Care?

A Neural Network created by an algorithm

Democratization of Deep Learning

The use of deep learning models is becoming more and more democratic every day and is becoming indispensable in many industries. Nevertheless, the implementation of efficient neural networks generally requires a background in architectural engineering and lots of time to explore in an iterative process the full range of solutions to our knowledge. The form and architecture of a neural network will vary in its use for a specific need. It is therefore necessary to design an architecture-specific to the given need. Designing these networks in a trial-and-error way is then a tedious task and requires architectural engineering skills and domain expertise. The experts use their past experience or technical knowledge to create and design a neural network. This implies that the set of architectures potentially used and evaluated will be reduced to those known by the expert.

Urgent need for efficiency and optimization

In some industries, the efficiency of a neural network will be fundamental. In order for neural networks to generalize and not overfit the training datasets, it is important to find optimized architectures. However, at a time when productivity is more important than quality, some industries are neglecting the efficiency of their models and are satisfied with the first model that achieves their objectives, without going any further. The search for suitable architectures is a time-consuming and error-prone task and requires architecture design skills. Due to lack of time or architecture expertise, these industries do not sufficiently exploit the potential of their data with “sufficient” models.

This article is intended to show the progress of the Neural Architecture Search(NAS), the difficulties it faces and the proposed solutions, as well as the popularity of the NAS today and future trends.

How to Understand the Complexity of Neural Architecture Search

Neural Architecture Search aims at discovering the best architecture for a neural network for a specific need. NAS essentially takes the process of a human manually tweaking a neural network and learning what works well, and automates this task to discover more complex architectures. This domain represents a set of tools and methods that will test and evaluate a large number of architectures across a search space using a search strategy and select the one that best meets the objectives of a given problem by maximizing a fitness function.

NAS is a sub-field of AutoML, which encapsulates all processes that automate Machine Learning problems and so Deep Learning ones. 2016 marks the beginning of NAS with the work of Zoph and Le or Baker and al, which achieved state-of-the-art architectures for image recognition and language modeling with reinforcement learning algorithms. This work has given considerable boost to this area.

Famous project

Neural Architecture Search (NAS) is one of the fastest developing areas of machine learning. A great number of research works concern the automation of the search for neural network architectures, in different industries and different problems. Already today, many manual architectures have been overtaken by architectures made by NAS:

• Object detection — Image Processing — Zoph and al 2017
• Image classification — Image Processing — Real and al, 2019
• Hyperparameter optimization — AutoML — Feurer and Hutter, 2019
• Meta-learning — AutoML — Vanchoren, 2018

Recent work on the NAS shows that this field is in full expansion and trend. 2019 and 2020 marks an acceleration in the amount of research being done. While early work could be considered proof of concept, current research is addressing more specific needs that cross several industries and research areas. This trend shows the potential that NAS can bring, both in terms of its efficiency and its ability to adapt to any type of problem but also in terms of the time saved by engineers to work on non-automated tasks.

The Benefits of Neural Architecture Search to Invest In or Not? What about limitations?

NAS methods explore a lot of potential solutions with variable complexities and hence are very computationally expensive. The larger is their search spaces, the more there are architectures to test, train, evaluate. These methods require huge resources and days to find a good enough model. Their reinforcement learning based NAS method Zoph et al. used 800 NVIDIA K40 GPUs for 28 days. Since the first methods, new models have emerged with much shorter search times. For example, Progressive Neural Architecture Search has demonstrated similar state-of-the-art results with 5–8 times faster search times. Efficient Neural Architecture Search takes about 7 hours to find this architecture, reducing the number of GPU-hours by more than 50,000x compared to NAS.

However, this area suffers from other several limitations. Indeed, it is hard to know how a potential model will perform on real data. As the architectures are evaluated with training data, the latter must be of good quality if we expect a performing model on real data. It remains necessary to define how the algorithm will find and evaluate these architectures. This task is still done by hand and needs to be fine-tuned. However, the lack of domain knowledge is not going to penalize in the efficiency of the architecture. This knowledge is useful to speed up the search process, it will guide the search and thus the algorithm will converge more quickly towards an optimal solution.

Recent algorithms such as PNAS methods try to approximate future performances but these predictors have to be fine-tuned and are still approximations. Additionally, certain approaches suffer from robustness issues and can be hard to train.

Some of the actual research now focuses on the use of biologically inspired algorithms as NAS methods. These algorithms are very efficient for optimization tasks and thus seem to be ideal candidates for finding the best architecture of a neural network.

To take away:

• NAS finds an ideal solution from a large set of candidates and selects the one that best meets the objectives of a given problem
• Optimization-based algorithms
• Biologically inspired
• Very Computationally expensive
• Hard to estimate how it will behave with real data

Conclusion

Neural Architecture Search is a rapidly expanding field in an era where optimization and performance are crucial. This very recent field still suffers from some difficulties to become a full-fledged step in the design of a Deep Learning project in industries. However, recent work tends to show that these difficulties will disappear in the coming years with the arrival of faster and more complete methods in the evaluation of architectures. The contribution of domain knowledge will no longer be indispensable but rather an advantage to improve the efficiency of research methods. NAS will thus bring more flexibility to industries and companies with these tools able to adapt to the plurality of specific needs.

🚀- Scarab's News

Thanks for reading this article! Leave a comment below if you have any questions.

References:

• Neural Architecture Search with Reinforcement Learning — Zoph and al, 2017
• Designing neural network architectures using reinforcement learning — Baker and al, 2017
• Meta-Learning, a survey — Joaquin Vanschoren, 2018
• Progressive Neural Architecture Search — Liu and al, 2018
• Efficient Neural Architecture Search via Parameter Sharing — Pham and al, 2018
• Regularized Evolution for Image Classifier Architecture Search —Real and al, 2019
• Hyperparameter Optimization — Matthias Feurer and Frank Hutter, 2019
• DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence — Byla and Pang, 2019