NHL Scouting Report Analysis: Are Defensive Prospects Overvalued?

Part 2 of my Text Analysis Series

This is the second post in a series where I dive into text analysis and how it relates to biases in the NHL draft. I added hundreds of Future Consideration's scouting reports to my dataset to dig deeper into NHL draft biases. Today we will use these scouting reports to see if NHL teams have overvalued defensively strong prospects.

What is a Draft Bias?

First, let's talk about draft biases and how to identify them. Again, I will use a methodology similar to the previous series mentioned in the introduction to find draft biases. You may want to skip this section for those following along with the series. For those new or interested in a refresher, this section is for you.

Note: This article will use Evolving Hockey’s Goals Above Replacement (GAR) metric. GAR's a catch-all metric measures how much a player contributes to their team’s performance. Higher GAR values represent more productive players.

The pick-value chart above gives us an expected value for each player based on their draft slot. Now, all we do is take the player’s cumulative GAR in their first seven seasons and subtract off expectations, and we have their GAR relative to expected.

For example, let’s look at Nail Yakupov. Yakupov was picked No. 1 overall in the 2012 NHL Draft by the Edmonton Oilers. As a result, his expected GAR from the chart is about 65. Unfortunately, he only produced about eight goals above replacement in the seven seasons after 2012.

As a result, Yakupov’s value above expected is -57 (8-65). In other words, He dramatically underperformed his draft position. Yakupov is a generational draft bust, so a large negative number for him checks out.

Now, all we have to do is apply this formula to every player drafted from 2007 to 2014 and start looking to see if any statistics can help us predict value above/below expectations. If certain players have been more likely to produce negative numbers, it suggests that NHL teams have overvalued those players because they generally fail to live up to their draft position.

Defining a Defensive Prospect

Now for the change from the previous series. I previously looked at measures like height to see if they were related to value above expected. Instead of statistical measures, we will look at qualitative measures taken from Future Consideration's scouting reports. I downloaded scouting reports for roughly their top 100 ranked players from the 2008-2014 NHL drafts. Then we will repeat a similar process to see if these players have been overvalued or undervalued.

Finding our two-way forwards will be slightly more difficult than finding power forwards because the terminology is more nebulous. We will look for key terms in the prospect's scouting report to group two-way prospects vs. the rest of the population. Those strings of words are; "well rounded", "responsible", "good defensive", "great defensive", "all around", "good two-way", "great two-way" and "complete".

Some prototypical examples of NHL players whose scouting reports contain these phrases include forwards like Brayden Point and Tomas Tatar, plus defenders like Ryan Murray and Aaron Ekblad. Some more surprising names include Steven Stamkos and Vladimir Tarasenko. Future considerations said Stamkos "plays a responsible defensive game" and Tarasenko "is responsible defensively and plays a solid three-zone game.".

I believe the NHL draft is about finding high-upside prospects. While a prospect being defensively responsible in junior certainly isn't bad, I have always wondered if the NHL has overvalued these prospects because of their perceived higher floor. On the other hand, a surprising number of elite NHL players like Point, Stamkos and Tarasenko were lauded for their defensive game as prospects. So now, we can determine if defensively responsible prospects have been overvalued or if the opposite may be true.

Are Defensive Prospects Over-Valued?

Now we can compare value relative to expectations for our defensively responsible prospects compared to the rest of the population.

At first glance, the opposite of what I had thought is true. Prospects with key defensive keywords in their scouting report have generally outperformed those without defensive keywords relative to their draft position. This is especially meaningful because we have a reasonably large sample of these defensively responsible prospects. However, unlike the previous post, this includes both forwards and defenders. It turns out this effect has been driven by forwards.

So for the remaining post, we will focus on forwards and note there are essentially no interesting findings around defenders. Additionally, note the relative sample sizes. There are significantly more forwards with these key defensive indicators, so we can be more confident in what conclusions we draw about the forwards.

Comparison To Power Forwards

Now that we are focusing on just forwards, we can compare groups using data from a previous article. Here we can compare the prospects who fall into our two-way forward cohort against the power forwards from this article to see some sharp differences.

Compared to the base rate of forwards, our defensive prospects have performed better on average relative to their draft position. Although this is a relatively small sample. The massive difference comes from comparing our Two-Way Forward cohort to power forwards.

Compared to our power forwards, two-way forwards have been far better draft picks relative to where they were drafted. As usual, this type of analysis can be sensitive to outliers, so we will look at success rates to add context to our small samples.

Success Rates of Two-Way Forwards

When using success rates,  each prospect is given either a one if they produced more GAR than expected based on their draft position or a 0 if they produced less. This way, no individual player can skew the results very much. What are the success rates of prospects across our different cohorts? Let's find out.

While the success rates offer a less stark contrast between our control group and the two-way forwards, the two-way forwards have still been successful picks for their draft slot a far higher percentage of the time than the power forward cohort.

Conclusion

It's important to keep stressing we don't have a massive sample size for this analysis. So throughout this series, I'm not trying to claim these questions have been definitively answered. Instead, I think these scouting reports can help us begin to answer questions that we otherwise wouldn't be able to test. This analysis acts as an exciting baseline rather than 100% accurate answers to these questions.

So now we know that I was wrong. Usually, I would be skeptical when I hear a prospect has been a great defensive player. Especially when I heard that about a forward. Of course, it's better to be good defensively than bad, but I thought these prospects got overvalued because the draft should be about finding upside, not defensively responsible prospects. While I still believe the draft is about finding upside, my intuition about two-way forwards was wrong.

In the past, Two-way forwards have been more likely to outperform their draft position than power forwards and the control group. Plus, some offensive superstars have apparently been defensive studs in their draft years. So while we can't be 100% confident two-way forwards have been undervalued, we can be confident they weren't overrated. The NHL has not been overvaluing these defensively sound forward prospects; if anything, the opposite is true.