How Fans and UZR Disagree: the Giants

When attempting to judge a player’s defensive skills, I prefer to look at a couple different metrics. Chief among them is the Fans’ Scouting Report (FSR) and Ultimate Zone Rating (UZR). Both are on the same scale, runs relative to average, but come to their conclusions in disparate ways. The FSR ratings come from surveys filled out by fans while UZR is entirely algorithm-driven. They take such different paths, but how close do they come to arriving in the same place?

One hundred and thirty-one non-catcher* hitters qualified for the batting title in 2009 and 127 such players in 2010. The correlation between the FSR and UZR was 0.56 in 2009. That rose to 0.60 in 2010. That is not a perfect match up, but a perfect match up would render them redundant anyways. For two systems that rely on such completely different inputs, I find that an agreeable level of correlation.

*UZR ignores catcher’s defense.

Just as agreement on a macro level is not interesting, so is general agreement on a micro scale. Precise agreement is worth a minor note. Hunter Pence is such a case as both FSR and UZR put him at three runs above average in 2010. Overall, however, I find it more interesting to look at the Andrew McCutchens, the Yuniesky Betancourts and the Matt Kemps of baseball. All three of whom had a greater than 15 run difference between FSR and UZR this last season.

Before looking at individual players however, I wanted to call attention to team-level ratings. While figuring out an individual player’s contributions are fraught with error and sampling issues, team defenses are much easier to gauge. Two teams stick out, and for opposing reasons, when looking at FSR and UZR: the Giants and Rockies.

In 2009, fans thought that the Giants defense was bad at -26 runs. They warmed to them in 2010, rating them as average, +1 run, as a unit. UZR has a very different interpretation, interestingly giving the Giants the same +56 runs in 2009 and 2010. In fact, the only Giant players to record a negative UZR with at least 100 innings of play at a position in 2010 were Aubrey Huff’s -1.3 runs in right field and Buster Posey’s -1 run at first base. The 82 and 55-run spreads between FSR and UZR for the Giants were the highest in the league each year.

Ignoring catchers makes this even a slightly bigger gap in San Francisco last season as Buster Posey was second on the Giants in FSR with nine runs while Bengie Molina was second worst with -5 runs and nobody had an opinion about Eli Whiteside other than that he has an awesome name.

The two biggest differences in FSR and UZR for the Giants come from Andres Torres and Edgar Renteria. Renteria is the biggest offender of the two. Fans rated Renteria well below average, a combined -21 runs for 2009 and 2010, but UZR saw the short stop as reasonably above average, a combined 2.7 runs. Andres Torres’ play is another example, garnering 29.4 UZR runs across the entire outfield but only 11 runs via FSR.

As big as the disagreement is on those two, the spread for the Giants does not come from a divergence of opinion on just those few players, but rather appears to be more of a systematic difference. Renteria and Torres totaled about 42 runs of difference between themselves, but the gap between FSR and UZR for the entire team is 137 runs.




Print This Post



Matthew Carruth is a software engineer who has been fascinated with baseball statistics since age five. When not dissecting baseball, he is watching hockey or playing soccer.

25 Responses to “How Fans and UZR Disagree: the Giants”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. Evan says:

    Perhaps people viewed the Giants as a collection of broken-down old veterans, none with a reputation for great defense? Even as a fan, I didn’t think our defense was nearly as good as the defensive metrics state, though I knew it wasn’t necessarily bad (Pablo Sandoval excepted).

    Vote -1 Vote +1

  2. As a giants fan, I must say I was surprised by how good the UZRs of Burrell and Sandoval were.

    Vote -1 Vote +1

    • Matt says:

      Burrell had the benefit of playing in left field in SF, where UZR is broken. Check out everyone who has playing in left field since the park opened and compare it to them playing in other places.

      Vote -1 Vote +1

  3. B N says:

    Two things seem likely here:

    1. Fans will tend to underestimate the number of runs by which a player’s defense matters. I would imagine this applies in both directions. I believe this is probably a product of being unable to “see” the total number of runs of impact. I mean… can you tell me how a 10 run defensive player differs from a 20 run defensive player? What you would look for? I can sit here and say “This player is twice as good as that one” but there’s no way to tell how many runs that amounts to, unless maybe you’re some sort of savant or you’re basically cheating by using stats to normalize your guesses.

    2. Renteria looks a lot less elegant than his stats indicate. I think it’s one of those issues where he does everything competently but nothing so great that you can point to it and say it’s great. Plus, he’s old. I think both of those factors would make people think he doesn’t play all that well. Finally, he’s got good defenders on both sides of him when he plays, since Uribe would typically be at 3B and Sanchez at 2B. That would likely reduce the range he has to cover, and I would imagine that his range is his worst quality as a SS (though still not bad, IMO).

    Vote -1 Vote +1

    • Xeifrank says:

      The fans in the FSR do not need to know how many runs a player saves on defense, that is all taken care of them by a formula. The fans only need to rate each player on a certain set of skills. Tom Tango (or anyone can do this) then calculates the runs saved (plus or minus) for each player based on the distribution of ratings at each position. So you only need to rate Pat Burrell with a speed of 20 and Juan Pierre with a speed of 70 etc…
      vr, Xei

      Vote -1 Vote +1

      • B N says:

        Ahhh, I see. That greatly clarifies this then, and should help with the magnitude. However, in that case I would be quite curious to see the underlying issues- i.e. where the fans assessments of skills differ with the outcomes used to estimate the runs. If I recall, UZR does consider various elements separately before combining them (i.e. arm is separate from fielding balls in zone). I wonder if there might be a way to determine the aspects where UZR’s components and the fans’ skills are really differing.

        Vote -1 Vote +1

  4. phoenix2042 says:

    does UZR change a defender’s zone when a shift is applied to the defense or is a zone always static? if not, then that can be a huge issue and make some players look really bad when a ball slides through the middle of their “zone” when they are shifted.
    also, i think that fans are hardly good judges of defensive prowess because a) they don’t necessarily watch every game/play, b) emotional attachment: they think more highly of players they like or penalize a player for one painfully timed error and c) offense can impact a fan’s idea of defense: ex. derek jeter and the gold glove.
    i don’t like UZR unless it’s over really good sample sizes usually. 3 seasons is a generally accepted sample size, which makes year to year WAR difficult to completely believe. i don’t think that defense should go back to fielding%, but i don’t completely trust UZR, FSR or TotalZone. generally, i like to combine them all to get a more accurate picture of a player’s defensive value.

    Vote -1 Vote +1

  5. Nate says:

    Something’s wrong with the fanscore for Torres, it was blatantly obvious from watching him that he was one of the best OF in baseball last year.

    Vote -1 Vote +1

    • Matt says:

      I don’t know if I completely agree with you about it being blatantly obvious. I agree that he’s a tremendous defender, but he isn’t what I would call a smooth defender. I think it might have something to do with his track style of running, but he’s always looked a little awkward out there to me. I eventually realized that it’s just the way he moves and it works for him (works great for him), but I can see people who might not have watched a lot of him playing thinking he’s a below average fielder who makes up for it with above average speed.

      Vote -1 Vote +1

      • Graham says:

        That’s an interesting possibility, but in general I, too, was shocked at how low Torres’ rating was. Maybe his speed in the field meant that he didn’t have as many Jim Edmonds “I’m not quite fast enough to make this look easy” catches, but Torres got to more balls than I can remember any Giant CF getting to over the past twenty years at least. Given how much of Triples Alley he had to cover at home, and the fact that he played at least half of Pat Burrell’s left field, Torres should’ve been way, way higher in FSR.

        Vote -1 Vote +1

  6. zenbitz says:

    I suspect that some of this might be due to Giants pitchers inducing weak contact/easy chances. The Giants had a great team UZR last year as well.

    Vote -1 Vote +1

    • Nivra says:

      This makes a lot of sense. Especially w.r.t. Cain. We know that Cain induces a lot of weak FB, and AT&T gives the defenders a lot of room to roam to catch those fly balls.

      Triples Alley did an interesting report regarding the Giants pitching and their extreme depression of IF grounders: http://triplesalley.wordpress.com/2010/12/28/on-the-giants-and-infield-defense/

      I wonder if the lack of IF grounders allows the IF to play deeper, and therefore allows the OF to play deeper, increasing UZR for the whole defense?

      Vote -1 Vote +1

      • Nate says:

        Yeah, I’m pretty sure that being the best strikeout staff helped out the numbers due to fewer total chances, but I also wonder about the effect of the shape of RF at ATT skews UZR which is dependent upon zones. Both by allowing more flyballs to be caught in deep RC that would be homers in other parks, but also because of the RF extreme positioning, who play well-off the line as it’s impossible to hit a tripe in the RF corner unless you get it stuck in the visitors bullpen bench.

        Maybe whenever Fieldf/X is released we’ll get an idea as ATT was the only stadium with it in 2010. It’d be great to know that Torres’s spectacular range was recorded for posterity.

        Vote -1 Vote +1

  7. SF 55 for life says:

    The thing that brings Torres down is his arm rating which really isn’t a plus, it’s just average. However his range is second to none right now. He glides out in center field and really does make it look easy.

    Vote -1 Vote +1

  8. Rob says:

    Am I right that the Giants led the NL in defensive efficiency last year? If so, I’m more inclined to believe the UZR scores in this case as the confirm the DE rating, which is really indisputable and likely a large enough sample to be accurate.

    Vote -1 Vote +1

  9. Nivra says:

    I found the FSR 2010 results link on The Book, but that isn’t converted into Runs like UZR. that seems like a 0-100 rating for various defensive skills. Can anyone point me to where I can find 2010 FSR converted into defensive runs?

    Vote -1 Vote +1

    • Nivra says:

      Nevermind. I found the FSR column on Fangraphs.

      Matthew, did you check the variance on FSR ratings vs. UZR ratings? The correlation may be .6, but the within-player variance may be much lower. That would mean a positively rated UZR team would get consistently underrated by FSR.

      Vote -1 Vote +1

  10. Steve says:

    Clearly the answer is Dodgers fans putting in bad ratings on Giants players. Ha, ha suck it LA.

    Vote -1 Vote +1

  11. Mr Punch says:

    I suspect that the issue with Renteria is that fans tend to judge fielders by exceptional plays (good or bad), whereas UZR looks at averages. Renteria rarely makes spectacular plays, and has stretches when he’s just awful. (The same is true of his hitting actually.)

    Vote -1 Vote +1

    • Fred says:

      Renteria was simply atrocious offensively in the postseason for instance.

      Vote -1 Vote +1

    • Heshan says:

      I think this similar logic can be applied to Torres. His speed actually hurt him with Gold Glove voting and fan ratings. His trajectory to the ball is different to the extent that he will beat the ball to the point where it will land and stop unlike many players who change their gait to reach the ball just as it reaches it’s landing point. (This is likely a horrible explanation but it explains why there aren’t as many spectacular catches by Torres as he makes those catches look like routine fly balls by stopping and waiting)

      Vote -1 Vote +1

  12. Bigmouth says:

    I don’t think it’s possible to overstate Torres’s defensive contribution last year. He covered a ridiculous amount of ground. In this case, UZR matches well with what I subjectively perceived.

    Vote -1 Vote +1

  13. The Grammar Nazi says:

    The Grammar Nazi wishes to speak to a pet peeve:

    “That is not a perfect match up, but a perfect match up would render them redundant anyways. ”

    The ‘s’ on the end of anyway is, uhhmmmm, redundant.

    Vote -1 Vote +1

  14. tangotiger says:

    The Fans Scouting Report had Torres at an implied +10 runs in CF, behind only Brett Gardner at +12 and Gutierrez at +11.

    Why do some of the commenters above say that there are surprised at how low the fans had him?

    Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current day month ye@r *