Models of You Perform Better than You

Daniel Kahnemin (author of Thinking Fast and Slow) has a new book about noise in decision making (aptly named: Noise). He describes the two aspects that contribute to error in human judgement as being bias (a consistent shift in judgement vs. the desired target) and noise (an inconsistent shift relative to a target, often random).

One of the ideas in the book is that simple mechanical rules outperform human judgment by eliminating noise. Human judgment is not so consistent. For instance, sentencing durations have been shown to be more lenient when the sun is out (likely due to the judge being in a better mood).

Another example study in the book centers around hiring decisions and job performance. In the setup, all you know is someone’s “score” on communication skills, technical capabilities, and leadership. Then you were asked to predict future job performance. A simple average of all the scores ended up beating the estimates of all participants in the study.

In a last example from the book, a model of someone’s predictions has been shown to be more effective in making decisions than the actual person. This is because the model of you is less noisy. In the example of sentencing decisions, taking judges decisions as outputs and various information about the case allowed for the construction of a machine learning model.

Sophisticated ML models, such as deep learning, also have the benefit of picking up on deep patterns in the data. The best ML models probably start to approach the limit of objective ignorance (the limit in what can be known given the information available).

The list below summarizes the methods of making prediction (ranked from best to worst, according to Noise):

1. Machine Learning Model of A Person’s Judgement
2. Simple Rules
3. Human Judgment

In the next sections I will attempt to apply these different approaches to Warren Buffett and his stock picks.

1. Machine Learning Model of Warren Buffett

To accomplish this, I created an ML model using Berkshire Hathaway’s holdings as the source labeling the companies. This previous article spells out how to web scrape a company’s holdings. I also include the link to GitHub for the code.

Using the Berkshire Hathaway holdings that were previously downloaded, we can use those to label quarters of companies that were held by Buffett. The trick was figuring out which other companies to include in the list as companies looked at by Warren, but ultimately, he decided to not buy. Short of reading his mind, I could only approximate this.

To start with, I planned on adding companies that I knew to be bad based on my previous experience with his methodology (GM and American Airlines). When I checked the list of holdings, GM and AA are both in there. What?!? Both have low profit margin, low income percent of revenue, and decent sized interest payments as a percent of earnings. It is possible that these picks are an example of noise (judgment deviating from a hard and fast rule). The other alternative is that Warren Buffet knows something that I don’t, which he definitely does.

Either way, I needed a different approach to labelling companies as “bad” Warren picks. I went with companies that failed his methodology the worst. I tried to get the ratio of training data setup so that ~20% of the companies were owned by Berkshire Hathaway and the other ~80% as “bad picks”. I am not sure this is the right ratio per se, but I know that there should be more bad companies than good in the data set.

I used 80% of the labeled data to build a random forest classifier model. I then ran validation on the remaining 20% of companies and the accuracy was close to ~90%.

Using this model, I then ran all the companies back through and averaged their scores across all quarters. I then picked the top 10 companies that had the highest average score. I made sure to pick companies across different sectors, in order to make the portfolio somewhat diversified.

Company Ticker List (BKNG, KO, COST, SPGI, MCO, JNJ, VRSK, V, PG, PAYX)

2. Modeling Warren Buffett with Simple Rules

One interesting application of this idea is, how would Warren Buffett’s judgments about stocks (equity holdings of Berkshire Hathaway) do against some simple rules that should model his behavior. In a previous article, I spell out the some of the rules used by Buffett for fundamental analysis.

I am going to reuse the performance of the stocks that I selected with those criteria. I documented the details of the selection in the article below. I used some ML to help with a rough ranking of companies, but ultimately the companies I selected meet the rules outlined in the article above.

Company Ticker List (ANSS, CME, IEX, ISRG, MA, MCD, MCO, MNST, PSA, SBUX).

3. Judgment of Warren Buffett Himself

This was actually the easiest one to come up with, because we can just use the value of Berkshire Hathaway as a rough approximation of the judgments made by Warren. For funzies, I included the S&P 500 as a benchmark of what the market did during this same period of time.

Company Ticker List: (BRK.B).

Output Comparison

Comparing all 4 approaches (1. ML model using Warren Buffett’s holdings, 2. Simple Rules that approximate his approach, 3. Berkshire Hathaway, 4. S&P 500) had some interesting results.

The first thing that jumped out at me is that Berkshire Hathaway didn’t beat the market over this period. I think if I went back another 10 years it would have, but unfortunately, I don’t have fundamental data or holding information from BRK to do this.

The second surprise is that the “simple rules” portfolio ended up doing the best. In hindsight, this makes sense. While the stocks in that portfolio all meet the fundamental financial rules warren uses, the companies were sorted using the ML model scores. This might make this category more representative of the “objective ignorance” than of simple rules. Looking back, I should have picked a different set of companies independent of their ML rating. The challenge here would be the time commitment to analyze enough companies to find good ones.

The last interesting point is that BRK is so close to the S&P 500 and the two ML models have such similar outputs. Probably half of BRK’s holdings are companies listed in the S&P 500, but Berkshire Hathaway holds way fewer companies than the 500 in the S&P. The only common company held between the two ML models was Moody’s. It ended up being the second or third performing companies in the list, but even when I dropped it, the portfolios still had very similar results. Maybe fact that they used mostly the same features led to them being so similar.

Conclusion

Hopefully by the end of this article you have a better understanding of how noise can impact decision making. I hope the approach from Kahnemin’s book Noise, applied to Warren Buffett gave you some good examples of the different approaches and their relative effectiveness. In the future I am considering using both ML models for evaluating stocks.

Note that this article does not provide personal investment advice and I am not a qualified licensed investment advisor. All information found here is for entertainment or educational purposes only and should not be construed as personal investment advice.