Learning From Mistakes That Were Made (By Me)

This post has been in the back of my mind for a while, and I couldn’t make it go away. And what better time to dust it off than for my first post of 2011? Although I’m just a hobbyist with too much time on my hands (I still can’t believe the Daves fell for it), I feel some responsibility to own up to when I’ve been wrong. Naturally, a full listing of stuff like that would go on far too long for a blog posting. Moreover, when dealing with stuff like statistical projections, one expects to turn out to be wrong a lot of the time (the trick is to be right, or at least “on the right side” a bit more often than not). Therefore, I’m not going to focus on posts I did that relied on projections that turned out to be wrong, but rather on three particular posts that I think were flawed in ways that I should have seen at the time. In that way, I can do more than just say “well, I guess that projection was way off” and actually take some lessons away from my mistakes.

1. ‘Garrett Atkins to the Orioles’, December 18, 2009. Going way back into the past, in this post I argued that the Orioles giving Atkins $4.5 million dollars guaranteed was okay because it was about his market value as something like a one win player. Of course, Atkins performed much worse than that, the dollar value of a win turned out to be much less than that once the free agent market settled, and the Orioles did not yet know that they would be in a position to sign Miguel Tejada at a relatively low price to play third base. Still, I should have been critical of the signing because it still really didn’t fit the Orioles. While given the projections and what we knew of the market at the time, the signing was “average” in the abstract, the actual situation showed something different. The Orioles weren’t going to contend in 2010, so there was no real advantage point in spending market value for a player who is a part-time/platoon player at best, Atkins was old enough that there really wasn’t enough upside, and finally, the Orioles already had a first baseman disguised as a utility guy by the name of Ty Wigginton at the time. The lesson learned (and I hope applied since then) was that I should going beyond just looking at how the wins/dollars match up and take more careful notice of the overall organizational context.

2. ‘Jeremy Guthrie as Baltimore’s Bright Spot,’ September 17, 2010. Back to Baltimore for the scene of the crime. In his post I basically wrote that Guthrie wasn’t as good as his ERA given his FIP and other peripherals. A number of people really disliked the post, and I can understand why they didn’t. It simply lacked in subtlety. I generally agree with Dave Cameron on why FanGraphs uses FIP for WAR, but in this post I focused so heavily on defense-independent stats that I didn’t sufficiently acknowledge how much we don’t know about pitching, such that that Guthrie might have the “knack” for outperforming his FIP. A better finish might have been to point out that while I wasn’t sure if he could continue to do so or not, that he makes an interesting case, rather than my more negative conclusion. The specific lesson is a reminder to be more careful when employing DIPs theory to analyze a pitcher. The more general lesson is to not be afraid to acknowledge areas of uncertainty and for further research, and to be willing to give open-ended conclusions to posts.

3. ‘Is a GM Gap Behind the AL/NL Disparity?’, January 4, 2010. Oh boy, this will get some laughs, and rightly so. Honestly, I think the overall idea was and is a good one. There is a talent gap between the AL and NL favoring the AL, and I do think part of it is that AL teams generally are run more intelligently (even if a bigger factor, as Rob Neyer pointed out, is that they also have more money as a group). However, the actual execution of the idea, which involved listing whom I then thought were the five best and worst general managers in baseball, was a bit lacking. First, as Neyer pointed out in the post linked above, focusing on the general managers as individuals rather than organizations as a whole (including the whole front office and ownership) is overly simplistic.

Second… well, just look at the “best and worst” lists and I’m sure you’ll have some laughs. I won’t get into defending or attacking each choice, even if people generally thought differently of Brian Sabean (to list just perhaps the most glaring example) prior to the season rather than after his team won the World Series. The one that really stands out to me for consideration now is putting Jack Zduriencik as one of the best. Again, this is a self-criticism based not on what turned out to happen, but on the information we had at the time. And at the time, I liked most of Jack Z.’s moves and the direction I thought he was taking the Seattle organization. But it was still problematic to put him in the “Top Five,” and would have been even if the Mariners had turned out to win 90 games in 2010 (something which I did not, by the way, expect). This isn’t a continuation of the #6 ORG discussion (go here, here, here, and here for that), but a discussion of where I went wrong in my listings, and the lesson here is that I forgot to apply one of the most elementary sabermetric (and statistical) principles: small sample size. I don’t mean in a literal mathematical sense, but in the more general sense that we had just a bit more than a year of Jack Z. and the rest of the new Seattle front office’s work on hand to judge him by. Even though I liked most of what I’d seen thus far, we’ve seen a lot of new GM’s come in and start making good moves that they’ve probably been thinking of making for a while in their previous positions, only to watch them flounder after a couple of seasons. I’m still not sure what will end up happening in Seattle or how good or bad Zduriencik and his staff are, and that’s the point: there wasn’t enough information, so if I was going to make such lists, Jack Z. shouldn’t have been on it. Small sample size even applies to fat front office guys (rhyming = great writing).

Whew. Hopefully this narcissistic exercise was a bit informative and entertaining. Again, keep in mind that this wasn’t meant as an exhaustive list of my mistakes, a list that will inevitably keep growing. To borrow a motif from Bill Simmons (boy, I hope I have enough “honor” to type this): the lesson is that I’m an idiot.




Print This Post



Matt Klaassen reads and writes obituaries in the Greater Toronto Area. If you can't get enough of him, follow him on Twitter.


34 Responses to “Learning From Mistakes That Were Made (By Me)”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. No way Simmons invented “I’m an idiot.” I’ve been saying that since I could talk, and I have a hunch people have said it in some way or another since speech was invented.

    Good post, Matt. I especially like the lessons learned from each post, as opposed to simply saying ‘I messed up’.

    Now if we could only get Alex Remington to write something like this about his NHL Winter Classic post…

    Vote -1 Vote +1

    • What I wanted to do with my Winter Classic post was apply an outsider’s perspective — the perspective of a baseball fan who doesn’t follow hockey — to current events in the NHL. I tried to make clear at the outset that I wasn’t well-informed, and I wanted to just write a think-piece about what struck me as a slightly absurd situation, constructing and then dismantling a rink for a single game.

      Obviously, it’s very hard to pull off a piece coming from a position of ignorance, and the point I tried to make probably wasn’t interesting enough to justify it — especially when, as commenters pointed out, the economics of a single-use rink are well-understood and the cost is not particularly egregious.

      That’s what I was trying to do, and, in my opinion, that’s why I failed.

      Vote -1 Vote +1

      • Paul says:

        Couple things. First, as one of the commenters critical of the article, I appreciate this explanation. Second, for what its worth, I think that a thought experiment garnered so much criticism because FG writers are largely held to a higher standard than some newspaper hack – and I’m not just talking about the embrace of statistical analysis. You’re an excellent writer, Alex, and I look forward to your next piece.

        Vote -1 Vote +1

  2. Locke says:

    The more general lesson is to not be afraid to acknowledge areas of uncertainty and for further research, and to be willing to give open-ended conclusions to posts.

    Bingo

    Vote -1 Vote +1

  3. Derek says:

    Thanks for going back and looking at, and admitting to, mistakes. I wish more places would do that.

    Cheers to you for learning from your mistakes and going forward with more knowledge instead of blindly forward like your 5 best GMs :)

    Vote -1 Vote +1

  4. Paul says:

    I like the point about small sample size, re: Jack Z. I think another way to say this is to have a healthy skepticism of conventional wisdom/what you read in the paper/what you heard in the bar. But not just accepting conventional wisdom applies to sabermetrics also. I think you just went with the crowd on the Jack Z. thing. I continue to be astounded at the reflexive conformity to anything that represents itself as saber or on a saber site like this one. That’s exactly the opposite of the point.

    Vote -1 Vote +1

    • T says:

      I don’t know, he had some legitimately amazing moves. It was only this year, mostly AFTER the organizational rankings and Matt’s posts, that he started to make some really questionable decisions.

      So the question becomes – do you let good moves influence your judgment even though it is a small sample, or do you wait it out before giving him a higher ranking? I think it wasn’t unheard of to trust he knew what he was doing, because his moves really were great AND there was information on him from Milwaukee that made it seem like he was legit. I don’t think anyone was wrong at all for putting him in the top 5 based on what they knew at the time, but it’s clear now he’s a fringe top 10 based on what he did this past year. And that’s results aside, since those were monumental collapses on an epic scale.

      Vote -1 Vote +1

    • jay says:

      The last thing I would charge the Fangraphs/SABR community with is “reflexive conformity”. The authors couldn’t write “2 + 3 =5″ without instigating howls of contempt and calls for their immediate firing. If anything, the opinions are over-the-top contrarian.

      Vote -1 Vote +1

  5. bc says:

    What’s best about this exercise is the way Matt tries to learn what went wrong with the prediction; not to just acknowledge the mistake. In re-reading the #6org aftermath linked above, after all that verbiage, it’s striking how DC cannot find or acknowledge a single error. But the willingness to subject your own mistakes to scurtiny to learn from them is what sets a scientific method apart from vapid polemic.

    Vote -1 Vote +1

  6. Danmay says:

    I think this is a great way to start of the new year. I, as a reader, expect the writers at this site to miss plenty.

    Sabermetrics, for me, is great for asking questions and learning new things. We often try to answer questions (like the NL/AL disparity) by trying to consider what relavent factors may have been improperly accounted for by previous analysis. Trying to account for everything within any system (i.e. baseball) is impossibly hard, no matter how much we try to simplify things that we think we know. Mistakes are expected, but so is learning from them.

    Vote -1 Vote +1

  7. DIVISION says:

    Isn’t it intellectually disingenuous to bring up Rob Neyer in a fangraphs article?

    If you read him on ESPN, you’d know that he uses no logic in making his arguments.

    He still isn’t over San Fran beating Texas in the WS.

    Vote -1 Vote +1

    • Jason B says:

      “you’d know that he uses no logic in making his arguments.”

      Damn Rob and his astrology and crystals. That charlatan.

      Vote -1 Vote +1

  8. JoeyPajamas says:

    Though I may be fairly knew to the SABR community (which I’m really loving, by the way), I can see how easily it is to make mistakes. Hell, players I used to think relatively highly of are nowhere near as good as I thought after some research and knowledge on advanced statistics. I can only commend you for putting yourself out in a public forum and admitting your mistakes and, of course, taking it further and learning from them.

    Matt Klaassen, you are a good, reasonable man. Excellent post, sir.

    Vote -1 Vote +1

  9. DrBGiantsfan says:

    Is it possible that Jack Z’s apparent commitment to acquiring players who scored highly on defensive metrics coupled with the fact that fangraphs.com champions these same metrics, might have led to some irrational exuberance on the part of some fangraphs.com writers re. Jack’s acumen as a GM?

    Is it also possible that the jury is still out on whether an emphasis on defensive metrics at the expense of offense is a successful way to build a team? I, of course, realize that the ideal situation would be to acquire players who are both good defenders and good hitters, but there seems to be a severe shortage of such players.

    Vote -1 Vote +1

    • phoenix says:

      a run saved is as good as a run earned. of course we can be accurate to quite a high degree in how many runs a player produces, but our ability to correctly calculate how many runs they save lags greatly. basically, wOBA is a hell of a lot more accurate and trustworthy than UZR/150 or any other defensive metric. i’d rather have player that adds 5 more runs than a player that saves 5 more runs because i don’t really know if he is in fact saving 5 runs. maybe it’s only 1 or maybe it’s 10, we don’t really know precisely, whereas we know exactly how many runs he has added to the second or third decimal. our knowledge of defensive measurements lags behind that of offense, so i put more stock in offense than in defense right now.

      Vote -1 Vote +1

      • DrBGiantsfan says:

        Jeff K,

        I think defensive stats like UZR can give you a rough idea of who is a good defensive player and who isn’t. Most of the time, we already have a pretty good idea of who those players are, but sometimes there are players everyone thinks are great defensively because they make spectacular looking plays, but really aren’t.

        If you are choosing between two approximately equal offensive players, you might favor acquiring the one who is better defensively, but you don’t pass up a clearly superior offensive player because the other guy might be better defensively.

        I think we are saying basically the same thing.

        Vote -1 Vote +1

      • DrBGiantsfan says:

        Oops,

        Meant to address that to Phoenix. My bad.

        Vote -1 Vote +1

  10. Jeff K says:

    Even from an extremely level-headed viewpoint, I don’t know how anyone could argue that Jack Z is not one of the best GM’s in the game. Yes, his team had the 2nd worst record in baseball last year, but it takes time to turn around an organization. And his team’s demise was brought on by 8 significant players have the worst years of their careers in the same season. The collapse of these players wasn’t age or past-performance related; it wasn’t predictable. I would think that a community like Fangraphs would understand that these types of statistical oddities happen every once in a while.

    If anything I would think that Jack Z solidified his reputation in the past season. Very few GMs pulled off trades last year as valuable for their teams as his flip of Cliff Lee. He traded a B prospect and two C prospects for one of the league’s best prospects and half a year of Cliff Lee. He absolutely deserves to be named as one of the best GM’s in baseball. It’s ok that some people are jumping off the bandwagon-there’s more room for us.

    P.S. I’m aware the writer is making an argument about sample size. My comment is aimed at those who think that the M’s performance last season reflects poorly on Jack Z’s GM skills, among other things.

    Vote -1 Vote +1

    • ppetterson says:

      Which 8 players are you talking about? If Wilson, then he’s never been worth anything at the plate. If Kotchman, the decline continued through Boston. If Bradley, he had a history of injuries. If Figgins, that move was widely panned at the time by the likes of Keith Law and others. He gave up a draft pick for a defense first 3B who moved to 2B and played poor defense (tons of errors). If Gutierrez, he was signed long-term after his first decent year, not exactly a long track record of offensive success. These are all low investment, LOW return moves. The only way it’s a good return is if what the computer says works out on the field with all that defense.

      Not to mention that the jury is most definitely still out on Smoak. If he doesn’t become at least a solid regular, he exchanged some prospects for a half year of a great pitcher to lose games in front of a terrible offense, and who was injured for the first month.

      I assert that no rational human being would argue that all this is either success, or not the GM’s fault.

      Vote -1 Vote +1

    • Locke says:

      I find it ironic that the original post is about admitting mistakes and this guy then writes the opposite.

      And I hate to break it to you, but winning 61 games in a season reflects badly on a GM, regardless of the cause.

      You can note good moves, of which there are many, but like every GM if you look hard enough there are a couple of clunkers. Brining back Griffey, signing Eric Byrnes and allowing him to make the 25man roster (ended up playing softball), League for Morrow, etc.

      Vote -1 Vote +1

      • Jeff K says:

        1) Do you think its possible to assemble an 85 win team that wins 61 games? (Do you even think its possible to know how good a team is before the season is over?)

        2) Are Jose Lopez, Chone Figgins, Casey Kotchman, Ryan Rowland-Smith, Milton Bradley, and Jack Wilson are going to play anywhere near as badly this season as they did last?

        Vote -1 Vote +1

      • Locke says:

        Yes, with injuries to star players it is very possible to put together a team on paper that should win 85 games which in reality only wins 61 games. Winning only 61 games with that much talent still reflects on the GM. There is no way to ignore this fact. The media/talking heads would blame the GM for picking up players with injury histories, i.e. he/she should have known. I’m not sure what you are arguing here, but make no mistake, winning 61 games reflects on Jack Z just like it reflected on the manager (whom was fired).

        I would feel very comfortable wagering that Lopez, Kotchman, Rowland Smith, Bradley and Wilson will together, perform as poorly, if not WORSE, than last year.

        Vote -1 Vote +1

    • DrBGiantsfan says:

      Jeff K,

      I was never all that impressed by Jack’s moves before last season unfolded, so no, I don’t agree that he’s a great GM who just go unlucky.

      Vote -1 Vote +1

  11. Resolution says:

    Props for this

    Vote -1 Vote +1

  12. Vince says:

    Is the AL “better” than the NL because they have more money; or do they have more money because they are “better”?

    Vote -1 Vote +1

  13. TexasRanger says:

    I forget who, maybe Dave wrote a post that people in the sabre community might tend to overrate players who use sabermetric principles to make themselves better. In the same vein we might also overvalue front offices who use these same ideas in team construction. Jack Z seemed to be a savvy statistician but there are others ways to construct a team, see Bill Smiths Twins. I believe that that might have had a hand in the Jack Z knob slobbing, by me as well, I was convinced that the M’s would challenge for the division.

    Vote -1 Vote +1

  14. Franco says:

    Omar Minaya mades some good early moves with the Mets.

    Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current ye@r *