The Beginner’s Guide to Measuring Defense

There’s a decent chance you’ve arrived at this page without a serious desire to hear more about defensive statistics. Trust me, I understand your frustration and your fatigue. Defensive stats like Ultimate Zone Rating and Defensive Runs Saved are controversial in some circles because they are reasonably new and the underlying data is somewhat hidden from view. You hear words like “flawed,” “absurd,” and “subjective” surrounding them. You’re tired of it.

Yet I’d like to lay out why we have advanced defensive statistics and how they work in the abstract. You won’t get to the end of this post and decide that UZR has perfectly measured Alex Gordon‘s defense, but hopefully you will have a better appreciation for why we measure defense the way that we do.

Why Errors and Fielding Percentage Come Up Short

For decades, the basic measurements of defensive performance were errors and fielding percentage. We used to judge players based on how well they avoided making errors on balls they fielded. If you didn’t kick the ball or let a throw sail wide, you were considered a quality defender. Certainly, those who were able to watch enough games developed a sense about which players were better defenders, but from a statistical standpoint, all we had were errors, assists, and putouts.

These statistics aren’t very useful, however. You certainly want to avoid errors because in order for something to be called an error you have, by definition, failed to convert a batted ball into an out. Yet there are two key problems with errors. First, they are determined by official scorers who don’t always make the right decision. Human error isn’t a problem, per se, but you’ve all seen enough scoring decisions to be skeptical about the quality of their decision making.

More importantly, however, is that errors are a subset of misplays. Even if official scorers got the rule book definition exactly right and perfectly uniform, we would still be ignoring a huge portion of bad defensive plays. Think back to a moment when you watched a player get a horrible jump on an easy ball. Think about the time an infielder took too long to get the ball out of their glove. Picture an easy pop fly falling four feet from the second baseman. None of those are errors even though they are relatively easy plays.

Measuring defense using Assists + Put Outs / Assists + Put Outs + Errors  ignores a huge slice of defense. If a player fails to get to an easy ball, there is no penalty. That alone should be reason enough for you to want something better.

Turning Batted Balls Into Outs

So if errors and fielding percentage fail to provide the entire picture of defense, what exactly should we be using? It’s not that errors lack importance, it’s that making an error isn’t the only way to screw up.

Instead of fielding percentage, the next step forward is something like defensive efficiency or Revised Zone Rating (RZR). Both statistics strive to tell you similar truths; how often a fielder turns batted balls into outs. In other words, we don’t care whether you make an error or if you don’t get to the ball. We care if you made an out or if you didn’t. The distinction between an error and a play not made is arbitrary. If 200 batted balls were hit to the third baseman’s zone during a given period of time, do we care if he made 20 errors and failed to get to 20 balls or if he made 10 errors and failed to get to 30? For the most part, we do not (unless they were horrible throwing errors leading to multiple extra bases). In both cases, he turned 80% of batted balls into outs.

This is a much better way to measure defense because it captures every play rather than just the subset of plays in which the fielder came in contact with the baseball. However, this type of metric has it’s limitations because it does not control for the difficulty or importance of the play.

Not All Batted Balls Are Alike

This is another simple truth about which everyone can agree. A rocket off the bat of Miguel Cabrera and a routine grounder from Seth Smith are very different batted balls. We want a defensive metric that includes all batted balls, but we also recognize that even moving in that direction doesn’t take use far enough.

Turning 80% of balls in your defensive zone into outs is great, but if a large portion of those are easy plays, that’s much less impressive than if they were more difficult.

Difficulty

A screaming line drive up the left center field gap and a routine fly ball to center field are both in the center fielder’s defensive zone. One of those balls is much easier to field than the other, so it stands to reason that more talented defenders would get to the more difficult play more often.

So we want a defensive statistic that does something to control for how challenging that particular play was to make. You should get more credit for a tough play than for an easy play. Typically, the modern defensive statistics (UZR, DRS, etc) measures this variable by determining how often that play is made by the entire league.

For example, if a certain play is made 40% of the time, then if the fielder makes the play, he gets credit for 0.6 times the run value of that play (we’ll get to this in a second). Because the average fielder should make that play 40% of the time, by making the play you get credit for the difference.

The advanced defensive stats all include these percentages based on multiple years of data. So if that screaming line drive is caught 30% of the time by center fielders, we’re basing that on all similar line drives over the last six seasons, for example. Humans have to code where the ball was hit, the approximate elevation, etc, but the algorithm is the one analyzing the data. The human being doesn’t say “that was a 40/60 play,” they say, “that ball was hit to X at about Y speed” and the computer compares it to all other similarly coded plays. There can be measurement error in defensive stats, but we aren’t talking about a random person guessing at the difficulty of the play.

Run Value

Using all plays and controlling for their difficulty is important, but we also want to consider how valuable it was to make that play. For example, imagine a hard hit ground ball deep in the hole at short. Maybe one out of every fifteen shortstops are able to turn it into an out. Call it a 7% chance the play gets made. If you make that play, you will get a lot of credit because it was very difficult, but if you hadn’t made the play, how much would it have cost your team?

In addition to difficulty, we also want to add in the average run value of the batted ball in some way. On that tough ground ball, if the play isn’t made then it’s almost always a clean single. It wouldn’t go for extra bases. No one is scoring from first. The play is hard to make, but the cost of failing to execute is lower than if we’re talking about a ball up the gap. So we want to multiply the difficulty times the value of making the play. You can read all about the specifics of this at our UZR Primer, but the concept should be pretty clear. You want the difficulty times the value of making the play.

So What Do We Have?

Instead of errors and fielding percentage, we want a defensive statistic that considers all batted balls and not just times the fielder touches the ball. We want a stat that measures the difficulty and the value of each play. That’s exactly what our advanced defensive metrics do. There’s nothing subjective or subversive going on here.

We’re taking some very fundamental questions about every play and we’re using multiple years of data to answer them. Limiting the number of errors you make is good, all else equal, but if you can get to 10% more batted balls than someone else while making a couple more errors, you’re almost certainly the better defensive player.

Limitations

It’s important to note that this does not mean that defensive stats are perfect. We’re relying on imperfect data. The video scouts can’t perfectly determine the location, velocity, and angle of every batted ball from watching the game tape. They do a very nice job, but there is measurement error. There will always be measurement error.

Additionally, sample size is an important consideration. There are a pretty small number of difficult batted balls hit to every fielder each year. If you luck into a few good plays or miss a few because you happened to be working with a bum ankle, your rating can fall quickly. That’s not a flaw of measurement, it’s a fact of baseball. You don’t get 700 chances to make dazzling plays each season. Even if we could get our measurements from an omnipotent baseball deity, we couldn’t do anything about sample size.

The same thing is true with offensive statistics. If a hitter goes 20 for 40 (a .500 batting average), you don’t say that his batting average is wrong. He got those 20 hits. What you might say is that 40 at bats is too small a sample to tell us very much about this hitter even if it is an accurate reflection of those 40 at bats.

Defensive metrics work the same way with respect to sample size. The metric isn’t wrong just because the output looks too large or small (although it could be wrong), but it might not be a very good reflection of what will happen in the future or how talented the fielder was for the previous few games.

You don’t need to take the precise measurements as gospel and I wouldn’t recommend it. But you should appreciate what these numbers are trying to tell you. These stats are answering the questions you want to have answered. There are all sorts of ways we might improve the measurements included in these stats and the ways in which we use them to determine talent and performance, but the fundamental logic is exactly what you want.

You want to know something about every ball hit to a fielder’s zone. You want to know how often that play gets made and if the fielder made it. And you want to know how valuable that play is on average. You don’t care about errors and put outs. You care about outs and things that aren’t outs. Don’t you?

Have questions about defensive stats? Ask them in the comments!



Print This Post

Neil Weinberg is the Site Educator at FanGraphs and can be found writing enthusiastically about the Detroit Tigers at New English D. Follow and interact with him on Twitter @NeilWeinberg44.


newest oldest most voted
Yirmiyahu
Member

With “Run Value”, do the advanced stats take into account baserunners/outs/game state?

Making an out on your hard-hit ground ball at SS is a lot more valuable when there’s a runner on 3B in a close game, than it is for that same ball with no one on in a blowout.

Referring to this component as “how valuable it was to make that play” implies that game state is taken into account, but that would mean we’re giving credit for defensive clutchness. We don’t want that. I’m guessing the algorithms just take into account how many bases that type of ball *typically* goes for if it gets through, rather than how valuable the out was in that particular instance?

Dan
Guest
Dan

What is the difference between the Def score in the team fielding page and the Def score that goes into team offensive WAR?

As an example, the Nationals show 34.3 Def score in the offense section and only 6.6 in the fielding section.

Yirmiyahu
Member

On the fielding section, they have -4.9 UZR runs, which somehow translates into a +1.1 UZR/150 (this seems like it’s a database error), and +6.6 runs from defense?

On the batting section, Fielding runs is +3.6, plus 30.7 runs positional adjustment, for a total of 34.3 total runs from Defense.

Cody
Guest
Cody

Is there any taking into account defensive shifts? How does that work? Sounds like a managers defensive shifts could severely alter the defensive value of players.

Cody
Guest
Cody

What about defensive positioning as well, sounds like that would matter a lot.

BaseballGuy
Guest
BaseballGuy

Seems obvious that good fielders help good pitchers, but isn’t it also true that good pitchers (especially those with excellent command) help fielders get better ratings because positioning works better when you know the pitch is going to be in a certain location. Can you tease out that effect?

jim fetterolf
Guest
jim fetterolf

Been suggesting this for awhile, Pitching Independent Fielding, PIF. Fielders are positioned and leaning for a specific pitch in a specific spot to a specific hitter. If the pitcher misses the defender is out of position.

AK7007
Member
AK7007

It’s counterintuitive, but Jeff looked a bit at this phenomenon awhile back. The Rays had the full lefty shift going in the infield, outfielders positioned normally, and pitches going to the outside third. They must be crazy right? When he looked at the numbers, it turns out that hitting a ball on the ground to the opposite field is very unlikely, even for pitches over the outer third. The batted ball profile on pitches to the outer third does change for balls hit in the air though. Positioning is more about knowing your own range limitations and the batted ball tendencies of the batter in the box, but not as much about where the pitcher is throwing. There might be some effect, but it’s not as big as those other factors.

Dennis
Guest
Dennis

Couple questions:

1: Who gets credit forruns saved due to a shift?
2: how is credit assigned on HR saving catches?

pft
Guest
pft

When are we going to get play by play, game logs and H-A splits for defensive metrics. Without these we as fans can not validate them and their use seems more a matter of faith than anything else.

BenRevereDoesSteroids
Guest
BenRevereDoesSteroids

Glad to see a nice new primer on defensive stats, but…

“For example, if a certain play is made 40% of the time, then if the fielder makes the play, he gets credit for 0.6 times the run value of that play (we’ll get to this in a second). Because the average fielder should make that play 40% of the time, by making the play you get credit for the difference.”

That part actually seems a little odd to me. Never actually knew that was part of it. Wouldn’t “Difficulty x Value” be akin to giving an extra boost to a hitter’s wOBA because his home run happen to come off Kershaw? Are there versions of UZR where you can just look at the Value or just a Difficulty average? Or am I just not grasping something?

Deelron
Member
Deelron

Liked the article, I’m definitely going to use “absurb” more often, it’s like absurd but it just has something else that makes it special.

joser
Guest
joser

As an urban guy, I find the asurbs are all around the edges of town.

Jdank
Guest
Jdank

Thanks for all this info. Here’s the question. How is an OF credited for cutting off a ball, cleanly hit into a gap or corner and holding the runner to first? Since there a hit, there’s no out recorded, but this defense has a huge impact on base runner position and subsequent plays. Thanks!

Andy
Guest
Andy

Don’t know if they do this, but they could just provide a run value for a ball hit in that area. E.g., if the ball is not cut off, it’s a double, which has a known run value greater than that of a single–about 0.30 run, I believe. Or there might be assigned a small probability that a ball not cut off would be a triple, which has a greater run value. One might determine a weighted value, based on the relative probabilities of a double or a triple resulting from a ball not cut off. So maybe somewhere between 0.30 – 0.40 run.

Then one determines from the historical data how often a fielder cuts off that ball. If it is 40% of the time, a fielder cutting off the ball gets credit for 60% of the run value, so let’s say, 0.18 – 0.24 run.

Emmett
Guest
Emmett

Does UZR account for bases saved on plays, even if an out isn’t recorded. Say a runner is on second and a hard ground ball is hit to the hole. Adrelton Simmons dives to stop the ball that only 15% of fielders get to, but doesn’t have time to get the runner at first. However, he stopped the runner on second from going home? Or when an outfielder cuts off the ball in the gap to hold a runner to a single? It seems like there is value here that might not be accounted for.

bstar
Guest
bstar

Emmett, I’m pretty sure DRS’s +/- system also functions as a sort of grab-bag to reward great plays that might fall through the cracks of what they are normally measuring.

Andrelton made a play last year by tagging out a would-be basestealer by reaching in between his legs to make the tag. I recall DRS using this as an example of a great play that they will reward.

Of course, he got the out on that one. So I don’t know about plays where an out wasn’t recorded.

Charles Mann
Guest
Charles Mann

Once Field FX and TrackMan A/S has more data, will the various groups be able to refine their defensive measurements? It seems that data from the tracking services will be more objective than a stringer’s subjective opinion on how “hard” a particular play might have been.

Bpdelia
Guest

I use the defensive metrics but they obviously are not perfect.

The effect of good range fielders next to each other is a serious problem (for instance Ellsbury and Gardner,Dyson and Gordon, etc.) but more importantly i don’t like that the adjustment is calculated with respect to the average fielder at a position. Gordon and Gardner should not be measured against only left fielders. It skews the averages too much.

I’d feel much more comfortable if the positional adjustment was simply OF, middle infield, corner infield.

Gardner shouldn’t look like a world beater because the Yankees happen to have a better defender on the team. He shouldn’t be compared with Carlos Quentin and the like

I think the stat would be much more useful if each defender was compared to a much larger group.

Also, no evidence, but the issue many seem to have is the WIN total being assigned to defense.

In his best years the fact that carl Crawford was worth three or four wins on defense alone was very difficult to believe.

Sometimes when data disagrees with conventional wisdom or common sense so drastically we need to closely examine if we’ve measured properly.

Otherwise we are just replacing one form of dogmatic belief with another.

magick sam
Guest
magick sam

what does seth smith have to do to avoid being used as an illustration of futility?

Bpdelia
Guest

Have a less alliterative name?

random internet dude
Guest
random internet dude

It used to be that UZR fielding %s were based on your position only, so if you had pop fly that the SS gets 90% of the time and the 2B gets 10% of the time (but is gotten 100% of the time) and the 2B gets the ball, then he gets the credit like he made a really difficult play when, in fact, he just made a routine play. Has that been fixed yet?

FIP'n good
Guest
FIP'n good

Awesome thanks!!!!!!!

Mike Pozar
Member
Mike Pozar

Is a middle infielder’s ability to turn double plays measured specifically?