Data Talk: A better way to grade individual players

Adam Ford

13 Apr, 2022

Data Talk: A better way to grade individual players

Welcome to the first edition of Data Talk, a series I made up in order to talk about data. We’re always working to improve the advanced stats we use in our content, so this post will cover two related topics: first, the introduction of a new metric for evaluating individual players, and second, a note that we now have past year’s data, so we can compare the 2022 Hogs to previous Razorback teams using the exact same metrics.

Grading individuals

So how do we grade individual players? When you’re wondering if a player is good or not, what do you look at? Obviously, raw stats like points per game and field goal percentage only get you so far. Points per game can be bad if the player takes too many shots to get his points, and field goal percentage isn’t too helpful if the player rarely shoots. And neither of those things takes individual defense or quality of opponent into account.

Real Plus Minus (RPM)

A few years ago, the rage became plus-minus, which I’ll call Real Plus Minus, or RPM. RPM tells us the scoring margin for a team while one of its players is on the court. So if Arkansas wins a game 68-60 and outscored the opponent 47-32 while Stanley Umude was on the court, then his RPM is +15. An RPM of +15 in a game you won by only eight means you did well… or at least, you were on the floor when the team was doing well. That’s the weakness of RPM: a high or low number could be total coincidence, since the individual contributions of each player aren’t being considered.

Plus-minus also doesn’t account for quality of opponent. In November 2020, KK Robinson was +30 in just 18 minutes against Mississippi Valley State. Because of that one performance — and because he played so few minutes the rest of the year due to injury — he actually led the Razorbacks in RPM per 40 minutes in 2021. But Mississippi Valley State was really bad: by our numbers, they were the worst team in the country.

Adjusted Plus Minus (APM)

We can fix the opponent-quality issue with RPM by converting it to Adjusted Plus Minus, or APM. All APM does is consider the Adjusted Scoring Margin of the opponent, and adjust plus-minus in that way. Example: Robinson’s +30 RPM against MVSU becomes +6 APM. What that’s saying is this: given the quality of MVSU, going +30 against them is the equivalent of going +6 against a perfectly-average Division I team. Against a good opponent, your APM can be much higher than RPM: Justin Smith was -3 against Baylor, but that translated to +16 APM. So assuming they played the same number of possessions in each game (they didn’t, but roll with it), we see that Smith was better against Baylor (+16) than Robinson was against MVSU (+6), even though Robinson was +30 and Smith was -3. That’s the power of opponent adjustments.

But APM still has many of the same problems as RPM: namely, it doesn’t consider individual contribution, so it could all be coincidence. Telling me that “Team A was really good while Player X was on the floor” doesn’t guarantee that Player X is good unless I know what Player X is doing out there.

So we need to consider a player’s individual stats.

Box Plus Minus (BPM)

If you want to integrate box score stats into your evaluation, then you need Box Plus Minus, or BPM. This formula is widely-used in the NBA is a way to evaluate players: writers use it to vote for all-star teams and scouts even use it to evaluate potential trades and free-agent pickups. The formula is very complicated — it occupies more than 300 lines of code in my scripts — but it does a lot: it determines a player’s position (the positions we use on this site, like “C” or “PG” or “SF”, are generated from the BPM formula) and his role (“creator” or “receiver”, or something in between) to create a perspective to evaluate all of his numbers.

BPM uses every available box score stat and spits out a big number for each player called contribution, which then has to be adjusted. This is where it gets dicey. The minutes-adjusted values for each player should add up to the total quality of the team, its Adjusted Scoring Margin. So the 2021 Hogs were +17.3 — basically, Arkansas was 17.3 points better than an average opponent per 100 possessions — so the sum of all minutes-adjusted BPM values should equal 17.3, since BPM is trying to tell us how much each player contributes towards those 17.3 points. To do this, the difference between the total team contribution and the team’s ASM is applied to each player’s contribution. This team adjustment is split evenly among all players, weighted only by minutes.

If that sounds complicated, here’s a simple explanation of the problem: just like RPM/APM don’t consider box score stats and thus have to assume that player is contributing equally to its numbers, the BPM formula doesn’t consider non-statistical things (like individual defense) and thus has to assume that each player contributes to those equally. But each player doesn’t: Au’Diese Toney was a great perimeter defender. There’s no stat to quantify that, so BPM doesn’t capture it. It assumes that Toney’s non-stat contributions per possession played are equal to, say, Jaxson Robinson’s.

So while BPM is good, it’s not perfect. In my search for something better than BPM, I came across the work of Evan Miyakawa, a data scientist who does some cool advanced stat stuff for college basketball. I recommend checking out his website. He’s created an advanced statistic that he calls BPR. He describes it here:

BPR quantifies the value of each player to his team on both offense and defense. A player’s ratings incorporate his individual efficiency statistics, along with his impact on the court for his team, which is assessed by looking at how successful his team was in every possession he played.

It sounds like he’s taken BPM and then evaluated some kind of plus-minus, thus effectively combining BPM and APM, allowing each to cover the other’s weaknesses.

Unsure of how to do something similar myself, I decided to try and create a statistic that complements BPM, by taking one more step with the plus-minus numbers.

Net Adjusted Plus Minus (NAPM)

I doubt I invented NAPM, but I created it in my data without any other inspiration. The concept is simple enough: if we can determine a team’s offensive efficiency, defensive efficiency, and scoring margin while a player is on the floor (that’s APM), then we can do the same thing for a when a player is off the floor, and take the difference to see how much better a player makes his team when he enters the game.

I debuted this stat during our season recap. Here’s what I wrote about the results at the time:

Outside of Vanover’s weird numbers, everyone is negative except for Notae and Jaylin, who sit at +24.7 and +23.4. These numbers are per 100 possessions, but the average game has about 70 possessions, so against a perfectly average team, if JD Notae didn’t play, Arkansas gets about 17 points worse. 17 points worse when Notae is out. The same holds for Williams.

Notae and Williams ranked 1st and 3rd in the SEC in NAPM. Arkansas was 24.7 points better per 100 possessions as a team when Notae was on the floor versus when he was off. That’s crazy. NAPM is a good measure of value in that regard.

But only after I published did I realize its major shortcoming.

I wrote in the recap that Devo Davis had a decent NAPM (4th on the team), while Kamani Johnson ranked dead-last among the 11 players who played enough minutes to evaluate. I said this makes Davis a “glue guy” (his BPM isn’t good but the team gets better when he’s out there). But is that really why his NAPM is good? I think there’s another explanation: when Kamani Johnson entered the game, who did he usually come in for? The answer, of course, is Jaylin Williams, who was second on the team in BPM. So NAPM is a zero-sum game: for Jaylin Williams to have a very high NAPM, the person who replaces him in the lineup must have a really low NAPM. Williams leaving the floor can’t make Arkansas 23.4 points/100 worse without causing his replacement to look really bad. So NAPM doesn’t really tell us how good Williams was: it just tells us that he was a lot better than Kamani Johnson. And NAPM doesn’t really tell us that Devo Davis is a glue guy: it just tells us that he rarely entered the game for Notae or Williams, and thus never took the big hits associated with replacing one of the two best players on the team.

So while NAPM isn’t the perfect measure I was looking for, it’s a great measure of indispensability. It tells you who a team cannot afford to lose. Teams with great depth will have everyone with NAPMs around zero (2021 Arkansas, where Devo Davis led the Hogs in NAPM at +3.3), while teams with poor depth will have some guys with high NAPMs and some with low NAPMs (2022 Arkansas).

So the search continued for the perfect catch-all metric.

FV Grade

Okay, so here it is. I’m calling it FV Grade for right now, although some generic name like Adjusted Box Plus Minus might suffice.

Here’s the basic theory: BPM is great, but it doesn’t account for non-stat contributions. So what if we found a way to make it account for those? Then we could adjust BPM to give us an accurate measure. Since using only same-team numbers (like NAPM does) creates a zero-sum game and compares players to themselves, we need to compare players to their opponents.

Imagine this lineup:

  • Arkansas: Moses Moody, Justin Smith, Jalen Tate, Devo Davis, Connor Vanover
  • Oral Roberts: Max Abmas, Kevin Obanor, Carlos Jurgens, DeShang Weaver, Kareem Thompson

Now let’s imagine that this group of 10 players was on the floor together for 10 possessions when the two played. We can formulate some kind of expectation for how those 10 possessions would play out using BPM. Arkansas’ five players have a cumulative BPM of +21.2, meaning they would outscore an average team by 21.2 points per 100 possessions on the floor together. If ORU was perfectly average, its 5-man lineup would have a total BPM of +0.0… but this ORU team isn’t average. Its five guys have a cumulative BPM of +6.1 (Abmas alone is +8.0).

Taking the difference, we see that BPM says that Arkansas’ five are 15.1 points per 100 possessions better than ORU’s five. But this 10-man lineup isn’t playing 100 possessions; for our purposes, it’s playing 10. So over those 10 possessions, we would expect Arkansas to outscore ORU by 1.51 points, given this lineup.

Now let’s say, in real life, Arkansas outscored ORU 10-8 during the 10 possessions those guys were out there together. Arkansas was +2 against an expectation of +1.51, so this lineup gets an adjustment of +0.49, since it did 0.49 points better than its expectation. ORU’s lineup gets a -0.49 adjustment for the same reason.

Once we have a whole game — or even a whole season — we can begin to apply these adjustments to BPM. For each player, we sum up the adjustment for every lineup they were a part of, and then divide by the total number of possessions they played to get Adjustment per Possession. We multiply by 100 (since BPM is per 100 possessions) and then add that adjustment to BPM to get ABPM. If the adjustment is positive, that player made contributions beyond the box score, and their ABPM will be higher than their BPM. In a sense, all we’ve really done is taken the standardized adjustment applied evenly to every player at the end of the BPM formula and weighted it based on how much better a player actually made his team.

I’m debating whether to keep ABPM or normalize it into a grade. Normalized values are 0-100, with 50 being average. A grade of 85+ is all-American territory, while 70+ is a good player and 60+ is a fine role player. Any player below 50 shouldn’t see minutes on a good team. I like the ease of understanding that the normalized values provide.

Applying FV Grade

Here’s how Arkansas’ 2022 players grade out using this new metric:

  • 86 – JD Notae
  • 83 – Jaylin Williams
  • 76 – Stanley Umude
  • 67 – Au’Diese Toney
  • 67 – Chris Lykes
  • 67 – Jaxson Robinson
  • 64 – Trey Wade
  • 59 – Devo Davis
  • 57 – Kamani Johnson

Notae and Williams were historically-good for the Hogs. For comparison, here are the top five players for last year’s team:

  • 79 – Moses Moody, 2021
  • 78 – Connor Vanover, 2021
  • 77 – Justin Smith, 2021
  • 76 – JD Notae, 2021
  • 71 – Desi Sills, 2021

The 2021 team had a ton of depth (five players 70+) while the 2022 team didn’t. Big surprise. I’m still working on the 2020 season, but I expect it will show that Mason Jones was really good (80+), Isaiah Joe was pretty good (70s), and everyone else was in the 50s and 60s.

Every advanced stat I can scrounge up severely overrates Vanover. He actually ranked 3rd on this year’s team in BPM and 1st on last year’s team. He was just 6th on the 2021 team in APM, but ranked 1st in APM on the 2022 team. Every number says he was productive and Arkansas was better as team when he was on the floor. His shortcomings were obvious — and his numbers were helped by the fact that he was often yanked early from games where it was clear he was overmatched — but I’m fascinated to see where he goes and how does at his new school.

Here’s how Arkansas’ incoming transfers graded out this season:

  • 71 – Trevon Brazile
  • 65 – Jalen Graham
  • 63 – Makhel Mitchell
  • 62 – Makhi Mitchell

There’s a ton of potential with this group, but it’s interesting to compare these names to others in the portal who drew interest from Arkansas:

  • 81 – Terrence Shannon, Texas Tech
  • 79 – Fardaws Aimaq, Utah Valley
  • 74 – Noah Carter, UNI (committed to Mizzou)
  • 74 – Eric Gaines, LSU (committed to UAB)
  • 70 – Brandon Murray, LSU (committed to Georgetown)

I have seen a little bit of negative chatter about the quality of the new adds (outside Brazile) given what was available. These numbers seem to back that up. Now you might be thinking that FV Grade alone isn’t good enough to determine future production. There’s probably something to that — after all, I literally just invented it in the last few days — but consider the 2-year FV Grades for this selection of transfers:

  • Stanley Umude: 75 at South Dakota, 76 at Arkansas
  • Au’Diese Toney: 67 at Pitt, 67 at Arkansas
  • Trey Wade: 64 at Wichita State, 64 at Arkansas
  • Vance Jackson: 57 at Arkansas, 58 at East Carolina
  • Desi Sills: 71 at Arkansas, 61 at Arkansas State

And for some non-Arkansas guys:

  • Xavier Pinson: 68 at Mizzou, 70 at LSU
  • Kevin Obanor: 69 at Oral Roberts, 79 at Texas Tech
  • Remy Martin: 79 at Arizona State, 79 at Kansas
  • Walker Kessler: 85 at North Carolina, 94 at Auburn
  • Oscar Tshiebwe: 66 at West Virginia, 94 at Kentucky

Tshiebwe is the only player with a change of more than 10 points. Six of these 10 had grades within two points of each other, despite being at different schools both year. Three of the 10 hit their exact same grade at their new school.

I really did not expect transfer players to be that consistent from one stop to the next. Among Arkansas’ transfers, I would give Trevon Brazile (who played just 42% of possible minutes this year) the best chance to significantly improve his already-solid grade. But the other three guys all played 60+% of team minutes, so unless there’s still some major development that Arkansas’ staff can help them unlock, they are what the numbers say they are. Maybe the staff has identified some opportunities with these players, but it’s interesting nonetheless.

I think this makes it even more glaringly obvious that Arkansas is well-set if Jaylin Williams returns, but in trouble if he doesn’t. If Williams comes back, then Graham and the Mitchell brothers would be competing for the job of backing Williams up, and all three would be a modest upgrade over Kamani Johnson. Brazile would be a modest upgrade over Trey Wade at the 4, and that’s before we even consider Jordan Walsh, who seems likely to start there on day one. But if Williams is gone, you’ve got three guys in the 60s trying to replace a guy who graded at 83. Not ideal.

It’s also possible that the movement isn’t done. Obviously, the Hogs are still waiting on Jaylin Williams’ NBA decision, but it wouldn’t be shocking if the Hogs see someone else hit the portal soon and then make another move if Williams also decides to leave. So keep your seatbelts fastened.