Summary

Deep learning methods are not recommended for time series forecasting unless one has extensive resources and expertise, as simpler statistical and machine learning methods often yield better results with lower costs.

Abstract

The article emphasizes that deep learning, despite its popularity, is not the most effective approach for time series forecasting for most companies. It cites a November 2022 study by Nixtla showing that deep learning methods not only underperformed compared to simpler econometric and statistical methods but also incurred significantly higher costs. The article advises against the common misconception that deep learning is superior, pointing out that claims of its performance are often biased or based on misrepresented data. It suggests that companies should focus on proven statistical, econometrics, and machine learning tools to build effective forecasting systems that deliver business value. The article also highlights a research paper from MIT where a simple benchmark model outperformed a sophisticated transformer-based architecture, reinforcing the argument that deep learning is not necessarily the best choice for forecasting tasks.

Opinions

Deep learning methods are often promoted by parties with vested interests, such as tech firms selling GPU hours or consultants.
The industry has examples of failed deep learning forecasting projects, suggesting that the method is not as reliable as traditional approaches.
Companies like Walmart, Target, and others have found more success with non-deep learning forecasting solutions.
Transformers, while effective in natural language processing (NLP), are considered fundamentally unsuitable for time series due to error accumulation and the inherent differences between language and time series data.
The article criticizes the practice of using only deep learning for time series forecasting, suggesting it is not a wise decision.
Even experts from Amazon's forecasting R&D team have acknowledged that deep learning may not be the best tool for the average forecasting use case.
The article implies that readers should be skeptical of claims made by deep learning proponents and instead trust empirical evidence and independent studies.
It is recommended to follow the author on Medium, Twitter, and LinkedIn for more insights into time series and forecasting.

𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐈𝐬 𝐖𝐡𝐚𝐭 𝐘𝐨𝐮 𝐃𝐨 𝐍𝐨𝐭 𝐍𝐞𝐞𝐝

Unless you have tons of clean data and tens of top PhDs working on forecasting for over a decade, as Amazon and Alibaba do, and even then, would you just take claims from the same companies that sell GPU usage for granted?

November 2022 update: in a large-scale independent study by Nixtla it was confirmed that Deep Learning methods failed to outperform a simple ensemble of econometrics and statistical methods whilst resulting in x25,000 higher cost (0.05 cents for econometrics/stats ensemble vs $11,000 of GPU costs for Deep Learning).

When it comes to time series and forecasting, “𝐃𝐞𝐞𝐩 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐈𝐬 𝐖𝐡𝐚𝐭 𝐘𝐨𝐮 𝐃𝐨 𝐍𝐨𝐭 𝐍𝐞𝐞𝐝.”

Do not let the deluge of papers about deep learning for time series (including ones produced by top tech companies and at top conferences) overwhelm you.

There is no evidence that deep learning methods, including transformers, outperform statistical and machine learning methods.

Most of the claims about the performance of deep learning come either from conflicted parties (including tech firms like Alibaba interested in selling more GPU hours) or academic labs that either by design or omission misrepresent the results of deep learning performance compared to other methods.

Such misrepresentation involves recycling the same toy datasets, dataset arbitrage (only showing results on datasets where deep learning works better), omitting non-deep learning benchmarks, not using correct benchmarks and many other tricks.

When starting the forecasting journey, don’t get distracted from fundamentals.

And whoever tells you that your company needs deep learning for forecasting is often either unfamiliar with the subject of time series or has a vested interest in selling an ineffective piece of consulting advice or a forecasting technology that does not work.

The industry is littered with examples of implementations of deep learning projects that result in ineffective solutions or complete disasters at worst, where deep learning systems exploded into production.

And unless coming from Amazon and Alibaba, one never hears about successfully implemented deep learning solutions, whilst many of the top companies like Walmart, Target, and others have either unsuccessfully tried and then abandoned deep learning or otherwise built effective forecasting solutions without deep learning.

As for your average Joe Bloggs Inc multibillion super-duper company listed on cool Nasdaq or not-so-cool NYSE stock exchange, deep learning 𝐝𝐨𝐞𝐬 𝐧𝐨𝐭 𝐰𝐨𝐫𝐤 𝐧𝐨 𝐦𝐚𝐭𝐭𝐞𝐫 𝐰𝐡𝐚𝐭 𝐲𝐨𝐮𝐫 𝐞𝐱𝐩𝐞𝐧𝐬𝐢𝐯𝐞 𝐜𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭𝐬 𝐨𝐫 not-so-deep in forecasting 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐏𝐡𝐃-𝐢𝐧-𝐢𝐫𝐫𝐞𝐥𝐞𝐯𝐚𝐧𝐭 𝐟𝐢𝐞𝐥𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐭𝐢𝐬𝐭 𝐭𝐞𝐥𝐥𝐬 𝐲𝐨𝐮.

Fire your expensive consultants and PhD-in-irrelevant-field Data Scientist (hello, Zillow) and save yourself a lot of time, trouble and millions in wasted project costs and foregone profits.

DO NOT use deep learning for forecasting unless you have tons of clean data, an expert team and a lot of time to play with these toys. Do not touch deep learning until you have built an effective forecasting system that delivers business value using statistical, econometrics and machine learning tools.

” FreDo: Frequency Domain-based Long-Term Time Series Forecasting”, a research 🧐 paper from MIT, pitted super fancy transformer architecture against a simple, almost mechanical benchmark.

𝕋𝕃;𝔻ℝ Transformer loses grotesquely

Let me repeat. A simple, almost mechanistic benchmark forecasting model from the Massachusetts Institute of Technology totally decimated sophisticated transformer-based architecture. And not one of your average transformers, the best and brightest of transformer for time series, the one that is better than another transformer, and so on.

𝕐𝕖𝕤, 𝕪𝕠𝕦 𝕙𝕒𝕧𝕖 𝕙𝕖𝕒𝕣𝕕 𝕚𝕥 𝕣𝕚𝕘𝕙𝕥. 𝕊𝕚𝕞𝕡𝕝𝕖 𝕒𝕝𝕞𝕠𝕤𝕥 𝕞𝕖𝕔𝕙𝕒𝕟𝕚𝕤𝕥𝕚𝕔 𝕓𝕖𝕟𝕔𝕙𝕞𝕒𝕣𝕜 𝕕𝕖𝕔𝕚𝕞𝕒𝕥𝕖𝕤 𝕥𝕙𝕖 𝕞𝕖𝕒𝕟𝕖𝕤𝕥 𝕥𝕚𝕞𝕖 𝕤𝕖𝕣𝕚𝕖𝕤 𝕥𝕣𝕒𝕟𝕤𝕗𝕠𝕣𝕞𝕖𝕣. Like a bunch of kids in Michael Bay’s movie won vs the meanest transformer 🍿

Simple mechanistic benchmark vs transformer

Like transformers? Stick to NLP. Transformers are excellent for 😎 NLP domain, just not as awesome for #timeseries. Not at all. Please don’t use them. Multiple research papers say that transformers are fundamentally unsuitable for time series for very good e never sees them winning forecasting and Kaggle competitions for good reasons.

It is not all about transformers. Using only deep learning for time series is generally not the brightest idea 💡. Even the most brilliant forecasting experts at Amazon Forecasting R&D fell into the trap of thinking that an average forecasting use case is amenable to deep learning. After the Kaggle Walmart M5 forecasting competition, they wrote a paper, “Learning with trees”, saying they finally realised deep learning is not the go-to tool for forecasting.

The reason transformers don’t work well for time series is straightforward — errors accumulate in transformer-based architectures, and there is nothing the big transformer can do about it. There are other good reasons, including that NLP differs from time series, so just because something works in NLP does not mean it would work in time series. Time series are not the same as word sequences; in sentences, only context matters; in time series order of the values is what matters.

Some consolation for DeepAR and others is that they are now in the good company of deep learning models for time series that do not work so well.

ARIMA boosted trees, …, secret sauce less known approach [hey, it’s a free Medium article did you expect freebie expert advice 🔑 ] > deep learning.

Want to learn more about time series and forecasting? Follow me on Medium, Twitter and LinkedIn.

References: