A Quiz

Last week Dave and MGL both made points about how just about anything can happen in 40 inning stints. To hammer that point home a little harder, let’s do an exercise that shows just how much variance exists. Below I’m going to list a few starting pitcher lines from the last 30 days; I’m also going to list the names of the owners of these lines, but not in order. Your job is obviously to attempt and match the line with the name without cheating.

A. 44.1 IP, 40 H, 4 HR, 8 BB, 21 SO, 2.23 ERA
B. 43 IP, 58 H, 8 HR, 4 BB, 36 SO, 4.4 ERA
C. 42.2 IP, 42 H, 4 HR, 7 BB, 52 SO, 4.22 ERA
D. 38 IP, 40 H, 9 HR, 8 BB, 32 SO, 4.97 ERA
E. 34.2 IP, 28 H, 2 HR, 13 BB, 28 SO, 2.08 ERA
F. 37.2 IP, 54 H, 6 HR, 7 BB, 12 SO, 6.21 ERA

Barry Zito
Mark Buehrle
Bronson Arroyo
Roy Halladay
Justin Verlander
Dan Haren

Answers after the jump.

Bronson Arroyo – 44.1 IP, 40 H, 4 HR, 8 BB, 21 SO, 2.23 ERA
Roy Halladay – 43 IP, 58 H, 8 HR, 4 BB, 36 SO, 4.4 ERA
Justin Verlander – 42.2 IP, 42 H, 4 HR, 7 BB, 52 SO, 4.22 ERA
Dan Haren – 38 IP, 40 H, 9 HR, 8 BB, 32 SO, 4.97 ERA
Barry Zito 34.2 IP, 28 H, 2 HR, 13 BB, 28 SO, 2.08 ERA
Mark Buehrle 37.2 IP, 54 H, 6 HR, 7 BB, 12 SO, 6.21 ERA

The best and worst pitchers alike can have identity crisis within 40 innings. If we judged them like a lot of people judged John Smoltz last week, then Barry Zito is the best pitcher in baseball and Mark Buehrle just isn’t cut out for the majors.

Print This Post

38 Responses to “A Quiz”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. El Guapo says:

    You’re comparing apples and oranges.

    Smoltz is coming off of shoulder surgery and he’s 42 years old. He refused to pitch from the bullpen despite concerns (rightfully so) that he could not carry the workload of a starting pitcher in the AL East. Not to mention his 40 innings represented 100% of his contributions this year.

    All the pitchers above have been healthy all year (and for the most part, the last few years) and their 40 innings, while certainly different from their career levels, only represent about 25% of their season numbers.

    And also, none of them had an ERA over 8!

    Vote -1 Vote +1

    • Nick says:

      The point is that you can’t tell much about a pitcher based on his ERA or even his FIP, because there is way too much variance in a small sample size.

      Even if Smoltz was severely hampered by the injury, and it’s possible that he was, you couldn’t tell that from a 40 inning sample because anyone can pitch poorly or well in that sample. If Smoltz had a 1.00 ERA, it still wouldn’t mean anything more than if he had a 8.00 ERA.

      Vote -1 Vote +1

  2. I’m inclined to agree with El Guapo. The general point about small sample sizes rings true, but there can be something new or different in some instances that indicates a true talent/skill level change. Nothing in the theories and explanations about the concept of small sample sizes implies that they are automatically noisy.

    Vote -1 Vote +1

  3. El Guapo is not accounting for true judges of talent though. Smoltz had a very good K/BB and was subject to a lot of bad luck in Boston. You need to show that his skills actually diminished and according to the things he can control like strikeouts, walks and groundballs he was just fine.

    How many pitchers in the history of baseball have had an ERA over 4.00 when they have had a K/BB of 4.67 and enough IP to remove luck?

    Vote -1 Vote +1

    • Nick says:

      Actually 6 pitchers have had an ERA over 4.00 with a K:BB > 4.66 and at least 100 innings pitched. The highest was David Wells, with a 4.45 ERA in 05, and the mean of the group was 2.71.

      Vote -1 Vote +1

  4. Matt B. says:

    Darn, got the first 3!

    Vote -1 Vote +1

  5. Phillies Fan says:

    In 2004, Johan Santana’s ERA for April and May (60 IP) was 5.61 and his ERA didn’t drop below 4.50 until he had pitched nearly 100 innings. I suppose that this means that, in order to satisfy the “small sample size” crowd, every pitcher that sets foot on a major league baseball field should be entitled to 100 IP before any judgment is made as to their abilities.

    But it just isn’t feasible for a team to allow every player a statistically significant amount of playing time to determine whether they can add value. So, in practice, a lot of judgment goes into determining when there is enough data to draw a conclusion on a player. A young first round pick, with great minor league stats, playing for a team that’s out of the pennant race will get plenty of time to show his stuff. A 42 year old coming off surgery playing for a team in the heat of a pennant race won’t.

    Did the Red Sox believe that there was no chance that Smoltz could still be effective? No. They made an educated guess based on a limited sample size, knowing that there was a chance that they were wrong, but believing that the cost of getting more data was too great. The Cardinals looked at the same data (along with somewhat different circumstances) and reached a different conclusion. Come October 4, we’ll have a better sense of who made the better bet.

    Vote -1 Vote +1

    • joker says:

      Again, they probably made exactly the same decision regarding Smoltz. He had been terrible but had a chance to be good based on stuff/numbers/everything, I don’t think the cutting edge Red Sox front office would miss the boat like that.

      The difference is their situations: Smoltz had massive incentive clauses with the Red Sox, he’s free for the Cards. The Red Sox have(had?) pitching depth which they thought was a better gamble, the Cardinals have Todd Wellemeyer. The Red Sox couldn’t even from a PR perspective trot him out in a pennant race, the Cardinals can. I can’t believe how this has gotten lost.

      Vote -1 Vote +1

      • Phillies Fan says:

        agree with all of that. by “reached a different conclusion” i didn’t mean a different conclusion as to whether Smoltz was a good pitcher or not, rather a different conclusion as to whether, all factors considered, Smoltz was worth the investment to increase their respective team’s chances at the playoffs.

        Vote -1 Vote +1

      • Joe R says:

        If the Sox missed the boat on anything, it’s the whole “pitch tipping” incident. If LaRussa had noticed that and no one in the Red Sox did, and Smoltz ends up crushing teams in September/October, heads should roll in the Red Sox coaching staff.

        Vote -1 Vote +1

  6. Other writers here, particularly Sarris in the fantasy section, have demonstrated a change in Smoltz’s slider for the worse. Troy is right that there were some positive statistical indicators for Smoltz in Boston, but scouting-based observations (which analysis of Pitch F/X seeks to appromixate) can carry substantial weight on a new level of performance even in a small sample size. New approaches or degraded talent can make even a 50 or less IP sample size significant.

    I don’t believe it’s so simple as to say all small sample sizes are wrong. All have greater chances of being luck based, but you might flip a coin twice and get a perfect 50/50 split.

    Vote -1 Vote +1

  7. rizzo says:

    Not sure what the point of the exercise is, I think we’re all comfortable with the concept of sample size. As has been pointed out none of the pitchers above is 42 years old and coming off shoulder surgery. My guess is that they would share Smoltz’s fate if those circumstances applied.

    Also – all are either signed to long-term contracts and would be very expensive to release or have approximately 10,000 times more trade value than John Smoltz.

    I’m far from a Boston fan, but it seems to me they probably explored all of their options and concluded that their choices were limited to leaving him in the rotation and praying he figured things out, or releasing him.

    Vote -1 Vote +1

  8. Sky says:

    When you have small sample sizes of data, you should look at finer-grained data. That means skip ERA in favor of K/BB. But also skip K/BB in favor of swinging-strike rates, pitch f/x movement and velocity data, and eventually hit f/x data.

    In other words, do with data what everyone should have been doing subjective in small sample size situations all along: trust the scouts. Of course, you need to trust the right scouts. The experts should be able to notice how Smoltz’s movement has changed, how his mechanics have changed, how his approach has changed. All those “subjective” things add up to produce results we can measure objectively.

    And again, we’re getting closer and closer to “objective scouting”. Pitch f/x is a good first step. Then hit f/x. Then eventually recording mechanics and crunching biomechanical data with computers.

    Vote -1 Vote +1

  9. Sky says:

    “As has been pointed out none of the pitchers above is 42 years old and coming off shoulder surgery.”

    This is an interesting piece of data. What do we do with it?

    Is a 42 year old coming off surgery with an 8.00 ERA and a 4.00 FIP more likely to be done than a 30 year old with an 8.00 ERA and a 4.00 FIP?

    Vote -1 Vote +1

    • rizzo says:

      Obviously there aren’t enough 42 year olds coming back from shoulder surgery for that to be conclusive, but why not take a look at all pitchers coming back from labrum surgery and see what their year 1 post surgery results look like?

      I have no idea what the results are, but i know what I wouldn’t do:

      I wouldn’t compare Smoltz’s sample size to Jered Weaver; Barry Zito; Mark Buerhle; Justin Verlander; Roy Hallday; or Dan Haren and use that as a reason to suggest that Boston erred in releasing Smoltz.

      I wouldn’t suggest that disagreeing with Cameron/Anderson = a lack of understanding of FIP or sample size.

      Vote -1 Vote +1

  10. James C-B says:

    All of these posts are dead on. Let’s be honest: no one who reads fangraphs would take that quiz and not expect Zito and Arroyo to be the 2 guys with ERAs in the 2s. Everyone here understands SSS, so you’re not eliciting any revelatory “WOW!”s from the crowd. Of course these samples are not predictive, because we have no reason to believe that anything – mechanics, makeup, velocity, etc. – has changed about these pitchers beginning 45 innings ago. However, small samples gain much more weight when you can predict them. For example, for John Smoltz, a 42 year-old coming off shoulder surgery, one could predict that he would be awful. Keep in mind that I’m not saying that his being awful was inevitable, just that it was a reasonable outcome and not a completely shocking result due to circumstances. On the other hand, for somebody like Halladay or Haren, the SSS argument is much more relevant, because you can say “well nobody saw that coming, nothing has changed about this pitcher, so these 40 IP constitute a statistical outlier and he will return to his career mean.” You absolutely cannot say that for Smoltz.

    Vote -1 Vote +1

  11. BrendanHarrisLives says:

    I bet you got the idea for this post from that guy in DRB who wrote the “Super Choke” article.

    Vote -1 Vote +1

  12. Alex JN says:

    blah blah blah shoulder surgery is missing the real point: everything we know statistically about baseball points to the fact that Smoltz’s astronomical era was mostly unlucky. Is he getting hammered? Maybe, but we can be almost positive his true babip with a line drive rate of 17% isn’t .390. Obviously the fact that those all fell for hits cost the Red Sox actual games, but you don’t make decisions based on what happened, you ignore sunk/costs/losses (except as performance in them alters projection) and you treat a player based on how he is going forward, and what we see from the skills we know to be. There is a valid point about what mean we regress him to, but it’s pretty ridiculous to make assertions about his true talent babip (and when you say he’s finished, that’s what you mean – his FIP is decent) from this small of a sample size (and, again, with a low line drive rate) when it has historically been so drastically more likely that a stretch of .390 babip this length was just noise.

    Vote -1 Vote +1

    • Marvin says:

      Yea smoltz horrible outing in boston was probably partially a product of bad luck and possibly tipping his pitches. But if you watched him pitch the first time through the lineup he was nasty old smoltz like, thats were his k/BB comes from even the second time through he wasn’t all that bad. But the third time he was leaving pitches in the middle of the plate and getting clobbered. That leads me to believe he was rushed back and didn’t have the stamina to start plus his location was still very rusty. call it bad luck his BABIP was so huge, I say it was also the batting practice fastballs he was serving up. Given the time off before and since he got to stl maybe/hopefully he corrected it. obviously boston didn’t think he could.

      Vote -1 Vote +1

  13. Rob in CT says:

    The question is not whether John Smoltz’s 8+ ERA was his true talent level – it probably isn’t. The question is whether John Smoltz’s true talent level was superior to the likely performance of the Red Sox’s other options: Buccholtz, Tazawa, Penny (since released), and the returning Wakefield and Matsuzaka.

    If Smoltz had been willing to hang out in the bullpen while Boston tried out Tazawa in the rotation, perhaps they would have kept him. He apparently wasn’t. Therefore, I see their decision as thoroughly defensible.

    Vote -1 Vote +1

    • Rob in CT says:

      Another thought: some of his poor results may be pegged to bad luck (high BABIP). How about bad fielding? The Sox are having a rough year in terms of defensive efficiency. Thus, I’m not sure you can expect a BABIP regression to league average. It’s probably going to remain higher (though not .390), unless you also believe that the Sox fielders have performed below their real levels and will be much better down the stretch (which I suppose is possible).

      Smoltz is now pitching in front of a superior (I assume) defense, facing weaker hitters. That seems like a recipe for improvement.

      Vote -1 Vote +1

  14. The Fonz says:

    Agreed with the majority of these posts…the funny thing is that I was able to get all these right because, in all the cases, the peripheral numbers (beyond ERA) were consistent with the player’s historical numbers. Which suggests to me that Smoltz doesn’t necessarily have to improve just because some of his peripherals are strong.
    That being said, St. Louis is incurring a “year of resurrections,” so it’s not out of the question.

    Vote -1 Vote +1

  15. Alex JN says:

    Well, something that could shed some light would be comparing his results in St. Louis with the results of the 4 and 5 slot pitchers in Boston. The peripherals thinkers already seem fairly convinced Smoltz remains a good enough pitcher to remain in Boston’s rotation; perhaps a good performance from Smoltz over the rest of the season would convince the results-oriented thinkers (those who think that his extremely poor results even given his solid peripherals justify thinking his future productivity is too low to make him a valuable part of the Sox rotation).

    Vote -1 Vote +1

    • James C-B says:

      Comparing results (and really, peripherals) in the AL East to those in the NL is a flawed exercise.

      Vote -1 Vote +1

  16. Pat says:

    I got them all!

    Did anyone else notice Dan Haren starts to tail off a bit in the second half? He’s an amazing pitcher and all but the last 2 season’s he’s been light’s out the first half, then a little above average the second half. I especially noticed this last year when he was on my fantasy team.

    Vote -1 Vote +1

  17. Nick says:

    For the love of god, drop the Smoltz BS. Smoltz just sucked in Boston, plain and simple. He had a high BABIP because his stuff was terrible.

    Deal with it, Fangraphs. Stop being a crybaby and just admit defeat.

    -13 Vote -1 Vote +1

    • Jon says:

      So did he also have a nearly 4:1 K/BB ratio because his stuff was terrible? (I honestly don’t know as I didn’t see him. Did he get a lot of strikeout that he shouldn’t have? “should” he have had more walks based on his stuff?)

      Vote -1 Vote +1

    • hazel says:


      Vote -1 Vote +1

  18. Phillies Fan says:

    what if both the “bad luck” crowd and the “pitch tipping” crowd are right. that means that Smoltz is so great that he can tip his pitches and, aside from bad luck, still pitch effectively.

    Vote -1 Vote +1

  19. Doug Melvin says:

    Yes, Smoltz’s 40 IP are of the exact same circumstances as those listed.

    I’m beginning to enjoy the weekly faux-outrage of this site.

    Vote -1 Vote +1

    • Nick says:

      Actually, his ERA in 40 innings is exactly as meaningful as those pitchers.

      Vote -1 Vote +1

    • Andrew says:

      All of the examples you gave are pretty bad. Barry Zito’s peripherals have looked really strong this season. He’s striking out a lot more people and inducing popups like he did back in Oakland. Mark Buehrle is very dependent on his defense so it’s not surprising his numbers would fluctuate. Plus there is the mental aspect of living up to having thrown a perfect game. A lot of people are saying Roy Halladay is injured and is #3 in pitcher abuse points this year (http://baseballprospectus.com/statistics/sortable/index.php?cid=465584). Justin Verlander was bad last year, #4 in PAP, and is #1 in pitcher abuse points this year. Dan Haren is always pretty bad in the second half, especially the last two years. His career ERA after the all-star break is 4.18 compared to 3.08 before it. He has 527 IP post-ASB so this isn’t a SSS issue. If any of you have him on your fantasy team next year, trade him immediately after the ASB. If anything, I think the pitchers you listed proves the validity of PAP.

      Shoulder surgery is a really serious injury for a pitcher. Plus Smoltz has had like 4 career-ending injuries. I really can’t think of anyone who came back from shoulder surgery and returned to dominate form when inserted back in the rotation. Finally, there were a lot of financial/opportunity cost reasons to release Smoltz

      Vote -1 Vote +1

  20. Pat says:

    Yeah, But what’s Nick Green’s ERA over the past 40 days?!

    Vote -1 Vote +1

    • Andrew says:

      Whoa it’s zero. It all makes sense now the Sox had to release Penny to make room for this hot pitching prospect.

      Otherwise the move makes no sense since Penny is being paid in big macs.

      Vote -1 Vote +1

  21. hazel says:

    Perhaps someone should write a post about statistical and PfX literacy of fangraphs readers.

    Vote -1 Vote +1