NHL Scouting Report Analysis: Are Intangibles Over Rated?

Part 3 of my Text Analysis Series

This is the third post in a series where I dive into text analysis and how it relates to biases in the NHL draft. I added hundreds of Future Consideration's scouting reports to my dataset to dig deeper into NHL draft biases. Today we will use these scouting reports to see if NHL teams have overvalued "intangibles" in the draft process.

What is a Draft Bias?

First, let's talk about draft biases and how to identify them. I will use a methodology similar to the previous series mentioned in the introduction to find draft biases. You may want to skip this section for those who read that series. For those new or interested in a refresher, this section is for you.

This has been done in a previous article which outlines the process behind this draft pick model (you can find that article here).

Note: This article will use Evolving Hockey’s Goals Above Replacement (GAR) metric. GAR's a catch-all metric measures how much a player contributes to their team’s performance. Higher GAR values represent more productive players.

The pick-value chart above gives us an expected value for each player based on their draft slot. Now, all we do is take the player’s cumulative GAR in their first seven seasons and subtract off expectations, and we have their GAR relative to expected.

For example, let’s look at Nail Yakupov. Yakupov was picked No. 1 overall in the 2012 NHL Draft by the Edmonton Oilers. As a result, his expected GAR from the chart is about 65. Unfortunately, he only produced about eight goals above replacement in the seven seasons after 2012.

As a result, Yakupov’s value above expected is -57 (8-65). In other words, He dramatically underperformed his draft position. Yakupov is a generational draft bust, so a large negative number for him checks out.

Now, all we have to do is apply this formula to every player drafted from 2007 to 2014 and start looking to see if any statistics can help us predict value above/below expectations. If certain players have been more likely to produce negative numbers, it suggests that NHL teams have overvalued those players because they generally fail to live up to their draft position.

Finding "Intangibles"

First, we should probably define intangibles. In this context, I am talking about the kind of intangibles often discussed in the NHL. For example, you will often see trades/signings justified by describing a poor hockey player as a great leader. Or maybe they are just a high-energy player, and the idea is their work ethic will rub off on teammates. That is the kind of player we are trying to find here.

To find prospects with these "intangibles," I have set up a program to search the few hundred scouting reports compiled for the following keywords; "leadership," "high effort," "competitive," "leader," "mature," "team player," "hard-working," "works hard" and "great attitude."

With his cohort of players, we can now check to see if players who had some intangibles reference in their scouting report exceeded their draft position or not.

So at first glance, we can see that players with some intangible reference have performed worse relative to their draft position than those without an intangible reference. So at first glance, these players possessing "intangibles" have been overvalued by the NHL draft market.

Next, continuing with our series on over and undervalued cohorts of prospects will compare the intangible cohort to the other cohorts identified thus far.

We have identified a pair of cohorts that produced less value than was expected, the intangibles group and power forwards. On the flip side, the two-way prospects stand out above these disappointing groups. So are intangibles overvalued? It is too early to say definitively, but the signs suggest they probably are.

Success Rates

I noted above we are working with small samples here. A common way I like to help ease this concern is by looking at success rates. Using success rates, each player can only receive a zero or a one. For example, think about Nail Yakupov and Johhny Gaudreau. Yakupov failed to live up to his draft position, while Gaudreau dramatically outperformed the expectations of a fourth-round pick. In the analysis above, these outliers may be able to skew the results in a small sample.

Instead, with success rates, Gaudreau is successful because he produced more than was expected based on his draft position, and Yakupov is unsuccessful because he did the opposite. How do the success rates look for the intangible cohort of players? Again, worse than the rest of the population.

Specifically, we can see that only about 21% of prospects with the intangible reference produced more than was expected based on their draft position. On the contrary, the group of prospects without an intangible reference outperformed their draft position slightly more than 27% of the time. And finally, we can compare the intangibles cohort to the other cohorts we have identified.

What About Poor Intangibles?

So we have been looking at the idea that NHL teams overvalue these “intangible qualities,” but what about the other side of the coin? What about players identified as having poor intangible qualities? To test this, I put some antonyms of the positive intangible works into the same system. Those words were “immature”, “cocky”, “selfish”, “poor body language”, “inconsistent”, and “lazy”.

Unfortunately, calling teenagers these words in a publically published report is uncommon. As a result, the sample is small enough this shouldn’t be taken as anything other than a fun fact. Nevertheless, the results were hilarious.

It turns out that prospects identified to have intangible issues performed significantly better than those identified as great leaders and whatnot. The difference was about 8 GAR, on average. Plus, the relationship held when using success rates too, so any individual outliers didn’t drive it.

Again small samples, so this isn’t definitive, but I thought it was incredibly interesting. It’s not like I think teams should draft bad people or anything, but maybe we over-rate our perceptions of which 17-year-olds are

Conclusion

In conclusion, I must again preach caution. Especially this time because intangibles are subjective. Plus, we will never be working with a massive sample here. Although it is worth noting the sample of players with intangible references in their scouting reports is relatively large.

So has the NHL overvalued intangibles in the scouting process? What I have collected suggests yes, they probably have. We can see this because players whose scouting reports referenced various intangible qualities like leadership and work ethic have been worse NHL players than you should have expected based on where they are drafted, on average. (Additionally, those identified to have poor intangibles performed far better than their draft position, on average).

This holds using binary and continuous measures of NHL success and suggests the players have historically been drafted higher than they should have, on average. So, we can't be 100% confident, but the best guess based on available information is that the NHL has probably overvalued these intangible qualities.