Improving pitcher projections

What if CHONE could scout pitchers?

What if the niftiest projection system out there could watch the pitchers throw, and make determinations how they would progress based on that?

Fangraphs’ pitch data gives us the tools to figure this out.

We’re starting from a more difficult place than projecting hitters. Projecting pitchers is hard. For one thing, they easily break. You look at Rich Harden funny, and his elbow pops. (Try it!) For another, they’re more erratic than hitters. So, we look at strikeouts and walks and BABIP and ground-ball ratio and ERA and possibly pet names. But so far, no pitch data. You can bring CHONE to a game and all it will do is complain about the TRS-80 you ported it to. It won’t watch.

But what if it could integrate scouting? What if it could add some numerical input based on scouty knowledge? Could we improve CHONE? Could we get substantially more accurate pitcher projections?

The answers, for those of you tired of the foreshadowing portion of the article and waiting impatiently for the content portion, are “We can,” “We can,” and “Yeppers.”

Here’s one scouty thing for pitchers: It is better to throw harder than softer. Announcers and some pundits like to say that it’s all about mixing pitches, changing speeds, hitting spots—and all those things are nice, and if you can’t do any of them, you’re Kyle Farnsworth—but it’s still better to throw harder. If over half your pitchers are fastballs and you throw 82, you’re playing against Sean Smith, not Albert Pujols, and I’m not the first to observe this.

My hypothesis was that pitchers who throw the ball real fast will do better than their projections. Pitchers who don’t throw the ball real fast will do worse.

So, if we had two pitchers:

Pitcher A, 5’11” 185 lbs, 165 IP 3.94 ERA 118K 74BB 1.71 GB/FB .299 BABIP
Average fastball speed 94.7 mph.

Pitcher B, 5’11” 185 lbs, 165 IP 3.94 ERA 118K 74BB 1.71 GB/FB .299 BABIP
Average fastball speed 85.2 mph.

Scouts would like Pitcher A a lot more.

Should CHONE look at the speed of the pitches? Does the speed of the fastball have a prediction value outside of the component stats?

I looked at the top 20 and bottom 20 pitchers in fastball speed for 2006. To qualify, a pitcher had to 1) Pitch at least 100 innings and 2) Use at least 50 percent fastballs (Tim Wakefield is not relevant to this discussion).

I then looked at their 2007 CHONE and their 2007 results. I did the same analysis for the following two years.

What I hoped to see was that the hard-throwing cohort would outperform CHONE, and the softer throwers wouldn’t. If this were true, this would be a significant breakthrough—the use of pitch data, breaks, and other numerically-definable scouting-type information could be used to generate some really excellent projections; improving on CHONE appears to be improving on the best projections available. If the hypotheses were correct, this would be the beginning of a new way to project pitchers—a better way.

Retroactive Review: Ace
Looking back at some of Justin Verlander's most interesting moments.

At this point, I must caution you, gentle reader. Unless you are some sort of Rotisserie or Scoresheet player, terrible pitching can hurt your eyes. Even reading about it may cause pain. The experienced fantasy leaguer knows that for every misery-inducing performance of one’s own team, there are many fabulously entertaining performances by one’s competitors. This year, those performances are called, “Rafaels,” after the Indians’ Mr. Perez.

Let us view those who try to get by with less velocity—but keep throwing fastballs. These lists start at the very slowest, and require that the pitcher throw at least 50 percent fastballs. The last column shows which was better, performance or CHONE:

2006 Slow-throwers     2007 CHONE    2007 Performance
                       IP    ERA       IP      ERA  CHONE or Performance?
Greg Maddux            197   3.93      198     4.14         CHONE
Livan Hernandez        211   4.79      204     4.93         CHONE
Mark Redman            180   5.06       41     7.62         CHONE
Kenny Rogers           190   4.40       63     4.43         CHONE
Aaron Sele             130   4.58       53     5.37         CHONE
Tom Glavine            192   4.18      200     4.45         CHONE
Kirk Saarloos          121   4.98       42     7.17         CHONE
Barry Zito             206   3.62      196     4.53         CHONE
Mark O’Connor          117   4.19    7.17 ERA in AA         CHONE
Jeff Francis           192   4.42      215     4.22      Performance
Paul Byrd              179   4.47      192     4.59         Even
Casey Fossum           147   4.88       76     7.70         CHONE
Chris Capuano          195   3.92      150     5.10         CHONE
Woody Williams         161   4.51      188     5.27         CHONE
Josh Fogg              172   5.17      165     4.94      Performance
Jason Jennings         180   4.05       99     6.45         CHONE
Steve Trachsel         140   5.15      158     4.90      Performance
Mark Hendrickson       172   4.46      122     5.21         CHONE
Pedro Martinez          63   3.11       28     2.57         Even
Zach Duke              195   3.94      107     5.53         CHONE

Three very marginal wins for performance. 3-15-2 is a pretty impressive set for the theory that low-velocity pitchers are bad bets, even compared to a good projection system like CHONE.

Will it get better if we look at 2008? Darken your sunglasses; the meltdown’s brutal:

2007 Slow-throwers     2008 CHONE    2008 Performance
                       IP    ERA       IP      ERA  CHONE or Performance?
Mike Maroth            120   5.48     Nada                  CHONE
Livan Hernandez        198   5.19      180     6.05         CHONE
Tom Glavine            194   4.78       63     5.54         CHONE
Lenny DiNardo          107   4.88       23     7.43         CHONE
Mike Bacsik            137   4.93     None                  CHONE
Barry Zito             202   4.19      180     5.15         CHONE
Greg Maddux            205   3.91      194     4.22         CHONE
Justin Germano         160   4.11       43     5.98         CHONE
David Wells            144   5.13      Called in Fat        CHONE
Matt Chico             166   4.93       48     6.19         CHONE
Paul Byrd              194   4.55      180      4.6         Even
Woody Williams         178   4.80     None                  CHONE
Orlando Hernandez      145   4.10     Nope                  CHONE
Jeff Francis           207   4.39      143     5.01         CHONE
Mark Hendrickson       143   4.28      133     5.45         CHONE
Noah Lowry             164   4.50     Broke                 CHONE
Chris Capuano          177   4.32     None                  CHONE
Andy Sonnanstine       184   4.50      193     4.38      Performance
Randy Wolf              98   4.04      190      4.3      Performance
Chuck James            155   4.41       29      9.1         CHONE

Even grimmer. CHONE collectively had some hope for these guys, but they were, as a class, even more thoroughly awful than expected; it’s like watching American Dad. If you rely on a fastball that isn’t fast, you’re going to get killed.

But maybe CHONE just has a sunny disposition when it comes to pitchers. (I think this is actually true; CHONE tends toward more optimism than some other methods, though it won’t be mistaken for the Bill James projections.)

Let’s go to the charts for the top-end velocity pitchers; the pitchers are listed from 1-20 amongst qualifiers in average speed – the last column again shows which was better, the projection or the performance:

2006 Best Fastball    2007 CHONE    2007 Performance
                       IP    ERA       IP      ERA   CHONE or Performance?
Felix Hernandez        177   3.34      190     3.92         CHONE
Justin Verlander       152   3.99      201     3.66      Performance
AJ Burnett             167   3.74      165     3.75         Even
Daniel Cabrera         172   4.19      204     5.55         CHONE
Josh Beckett           186   4.00      200     3.27      Performance
Scott Proctor           91   3.99       86     3.65      Performance
Brad Penny             179   3.77      208     3.03      Performance
Seth McClung            99   4.98       12     3.75         Even
CC Sabathia            195   3.58      241     3.21      Performance
Matt Cain              184   3.50      200     3.65         Even
Jeremy Bonderman       200   3.63      174     5.01         CHONE
Kelvim Escobar         154   3.69      195     3.40      Performance
Chien-Ming Wang        182   4.09      199     3.70      Performance
Ervin Santana          169   3.94      150     5.76         CHONE
Johan Santana          212   2.51      219     3.33         CHONE
Jorge Sosa             119   4.20      112     4.47         CHONE
Ian Snell              169   3.94      208     3.76      Performance
Roy Oswalt             211   3.28      212     3.18         Even
John Smoltz            191   3.59      205     3.11      Performance
Ben Sheets             155   3.01      141     3.82         CHONE

OK, pretty close: 9-7 in favor of performance. But this is a decent set for the hypothesis.

Let’s see how our hard throwers did in 2008:

2007 Best Fastball     2008 CHONE    2008 Performance
                       IP    ERA       IP      ERA  CHONE or Performance?
Felix Hernandez        191   3.53      200     3.45         Even
AJ Burnett             165   4.04      221     4.07         Even
Justin Verlander       188   3.88      201     4.84         CHONE
Dustin McGowan         157   4.41      111     4.37         Even
Josh Beckett           197   3.79      174     4.03         CHONE
Daniel Cabrera         195   4.57      180     5.25         CHONE
Tim Lincecum           118   3.28      227     2.62      Performance
Edwin Jackson          153   4.76      183     4.42      Performance
Zach Greinke           124   4.35      202     3.47      Performance
Kelvim Escobar         175   3.86      None, thanks         CHONE
Fausto Carmona         180   3.85      120     5.44         CHONE
Jeremy Guthrie         155   4.76      190     3.63      Performance
Brad Penny             195   4.11       94     6.27         CHONE
Matt Cain              196   3.54      217     3.76         CHONE
CC Sabathia            221   3.50      253     2.70      Performance
Ben Sheets             137   4.01      198     3.09      Performance
Roy Oswalt             213   3.59      208     3.54         Even
Chien-Ming Wang        189   4.33       95     4.07         CHONE
Matt Albers            153   5.53       49     3.49         Even
Jake Peavy             217   2.99      173     2.85         Even

Alas, a 6-8 run. When I did this for PECOTA, the numbers had a much better showing for the performance than the projections. CHONE’s general optimism for pitchers may well be because it’s right to be more optimistic than other systems.

Now, I did these charts after the fact, and I made the prediction that 2009 would be similarly situated. Was my method able to predict the future? Let’s go to the slowest of 2009 (with the prior caveats; retired pitchers weren’t considered for this year.)

2008 Worst Fastball    2009 CHONE    2009 Performance
                       IP    ERA       IP      ERA  CHONE or Performance?
Livan Hernandez        172   5.23      183     5.44         Even
Barry Zito             176   4.70      192     4.03      Performance
Jeff Francis           169   4.58      143     5.01         CHONE
Jeff Suppan            161   5.37      161     5.29         Even
Chris Young            114   3.95       76     5.21         CHONE
John Lannan            148   4.68      206     3.88      Performance
Jarrod Washburn        151   4.41      176     3.78      Performance
Greg Smith             144   4.88    Minors/bad/hurt        CHONE
Pedro Martinez          76   4.45       44     3.63         Even
Scott Olsen            176   4.96       62     6.03         CHONE
Mark Hendrickson        60   4.20      105     4.37      Performance
Brian Burres           127   5.24   Mostly minors/bad       CHONE
Brandon Webb           209   3.70          Ow               CHONE
Chris Sampson           72   3.88       55     5.04         CHONE
Darrell Rasner          96   4.50      113     5.40         CHONE
Nate Robertson         171   4.53       49     5.44         CHONE
Andy Pettitte          166   4.55      194     4.16      Performance
Garrett Olson          157   4.24       80     5.60         CHONE
Tom Gorzelany          152   4.32       47     5.55         CHONE
David Bush             180   4.30      114     6.38         CHONE

Best year ever for our soft throwers, going 5-17-3, and with pretty big wins by Zito and Lannan. While not reflected in the charts above, the data suggests to me that lefties should receive about one mile per hour credit for being lefties; it’s actually true that left-handers can get by with less velocity.

Let’s see how our hard throwers did in 2009:

2008 Best Fastball     2009 CHONE    2009 Performance
                       IP    ERA       IP      ERA  CHONE or Performance?
Joba Chamberlain       102   3.26      157     4.75         CHONE
Ubaldo Jiminez         166   4.34      218     3.47      Performance
Dustin McGowan         118   3.97         Broke             CHONE
Felix Herndandez       185   3.60      238     2.49      Performance
Ervin Santana          191   3.77      139     5.03         CHONE
Josh Beckett           177   3.61      212     3.86         Even
AJ Burnett             167   3.83      207     4.04         Even
Tim Lincecum           154   3.21      225     2.48      Performance
Clayton Kershaw        115   4.15      171     2.79      Performance
Edwin Jackson          145   4.78      214     3.62      Performance
CC Sabathia            211   3.41      230     3.37         Even
Justin Verlander       185   3.94      240     3.45      Performance
Edinson Volquez        166   3.58       49     4.35         CHONE
Johnny Cueto           145   4.19      171     4.41         Even
Zack Greinke           137   4.14      229     2.16      Performance
Jeremy Guthrie         156   4.33      200     5.04         CHONE
Seth McClung            68   3.84       62     4.94         CHONE
Matt Garza             168   3.96      203     3.95         Even
Jorge de la Rosa       108   4.67      185     4.38      Performance
Fausto Carmona         129   4.26      125     6.32         CHONE

We see a slight edge here for the performance, but it’s not big – further study is needed to see if there’s a reliable effect. The 2009 crew of hard throwers had little more injury-related falloff than prior years.

For now, we can confidently assert that slow-throwers who rely primarily on a fastball will do worse than their prior component stats and worse than a sophisticated projection model would otherwise predict.

We can unconfidently assert that fast-throwers will do slightly better than their prior component stats and better than a sophisticated projection model would otherwise predict.

This is just the first step. We can look at how the pitch selection, and pitch speeds affect likely outcomes. At some point, we can incorporate pitch break. We can improve CHONE. Or, someone can improve CHONE who has actual database skills. Trained monkeys have better database skills than I do (at least, the trained monkeys who generate Marcel at Tango Palace do.)

I’ve taken a few additional steps in modifying the formula: [Velocity=Good]. Because I’m a reprehensible human being, I’m not sharing. But, here’s the list of leaders and trailers for next year, under the alpha version of the projection modification system. This is a ranking of *differences from an established projection system*, not overall goodness. If you go draft Jeremy Guthrie ahead of Tim Lincecum, you’re missing the point.

1. Homer Bailey (You can imagine my concern about putting a Dusty pitcher here.)
2. Ubaldo Jiminez
3. Edwin Jackson
4. Justin Masterson
5. Jeremy Guthrie
6. Justin Verlander
7. Brad Penny
8. AJ Burnett
9. Josh Johnson
10. Felix Hernandez

1. John Lannan
2. Jered Weaver
3. Aaron Laffey
4. Derek Lowe
5. Dallas Braden
6. Brian Moehler
7. Jeremy Sowers
8. Zach Duke
9. Joel Pineiro
10. Ted Lilly

I expect the Good guys to have ERA’s about 5 percent less than CHONE projects, but I could be wrong. I expect the Bad guys to pitch many fewer innings than CHONE expects, and be far less effective, and I’m not wrong on that.

At the end of the season, we’ll revisit if Dave Studeman drops the restraining order and lets me stuff more crumpled charts on yellow lined legal paper on his doorstep.

What do you think? Have I missed something important? Or am I right?

Place your bets in the comments.

Print This Post
Sort by:   newest | oldest | most voted
Sean Smith
Sean Smith

For 2009, the hard throwers had a simple average of 3.94 projected, and 3.94 actual.  Weighted by innings, they beat the projections 3.75 to 3.91.

For the slow throwers they were worse 4.96 to 4.52 on simple average, 4.79 to 4.58 weighted average.

But I should go a step further, weight both projected and actual era by actual innings pitched.  Give me a few minutes.

Wow! Comments from MGL. I am honored. I know that the keeping-score method isn’t any good in close cases. The bad pitchers aren’t close cases. It’s more visually impressive to do it this way. I used a method for which was “better” based partly on value; substantially more innings of similar quality is better. Pedro had far fewer innings while Hendrickson had substantially more; that’s why I did this. (In both cases, they were on the thin edge of the margins using my method, and I preferred to count them against my theory.) So, in short: 1. You’re right, and… Read more »
Mike Fast
Mike Fast

Excellent work, John!  This is obviously worthy of further investigation.

Sean Smith
Sean Smith
I took both projected ERA and actual ERA, and weighted both by actual 2009 innings pitched.  Doing that the hard throwers beat their projections 3.75 vs 3.95.  And the soft-tossers post a 4.79 ERA, compared to 4.61 projected. Great work by John, and I’ll take a little credit for suggesting he contact Studes and publish on THT. The next question is how/why do they beat the projections?  Do they strike more out?  Allow fewer homeruns? I did something similar to this two years ago and found no effect: The differences between what I did and John did are twofold:… Read more »
Nick Steiner
Nick Steiner

Great job John.

I edited out about three hundred words at the end; things that might be of interest to the onlookers, or not: A. The first version of this actually used PECOTA, back when PECOTA hadn’t been ground into a fine powder. I liked that study better, because PECOTA’s more pessimistic toward pitchers generally, and I ended up with robust data on both sides (I don’t care if it was more accurate; I want it to match my hypotheses, dammit!) On a private board I’m on, someone suggested that maybe this shows a problem with CHONE rather than an effect by slow-throwers.… Read more »
Interesting work.  One concern:  this could be telling us only that CHONE over-regresses very good and very bad pitchers (at least those healthy enough to qualify for the study), as opposed to velocity.  If you simply took the 20 best and worst projections for starting pitchers and compared them to actual performance, would we see the same divergence?  (Great pitchers outperforming CHONE, bad pitchers underperforming).  A suggestion:  first adjust the projections to equal the actual outcomes in the aggregate.  That way the comparison will be much clear.  (And we don’t care, for these purposes, about the accuracy of CHONE’s league-wide… Read more »
Sean Smith
Sean Smith

For the 2009 list of pitchers, as a complete group my ERA projections were almost dead on once you weight both the projections and the performance by actual innings pitched.

I was .20 too high on the fastballers, almost perfectly matched by being 0.18 too low on the soft-tossers.

“The idea that Sean Smith couldn’t find a correlation scared the hell out me”

Shouldn’t, as my study was not a complete one and looked at a different subset of pitchers.  Sometimes if you want to find an effect you’ve got to look under every rock.  I just looked under 1.

Sean Smith
Sean Smith
It would be easy to check the best 20 starters in projected ERA and see how they match with reality, and see if I’m regressing enough. Identifying the 20 worst is not so easy, since the projection list includes guys who were hammered for 50 innings in AA and have a projected ERA of 7.50 or something, most of whom won’t pitch in the majors.  And if you look for the guys with the worst projections who were actually allowed to pitch, well, that would be some serious selection bias.  My guess is you could find a group with projected… Read more »
Sean Smith
Sean Smith

I combined 2007-2009 best fastballs lists.

Weighted by actual innings, projection is 3.85, actual ERA is 3.79

For the slow throwers, 3 years data, projection of 4.47, actual of 4.94.


Im not sure how effective this analysis will be without distinguishing between 2 and 4 seam fastballs.  The pitches have significant differences in profiles despite their similar speeds

Sean Smith
Sean Smith

“One concern:  this could be telling us only that CHONE over-regresses very good and very bad pitchers”

This is not the case, at least for the good pitchers.  I took the top 20 projected ERAs for starters last year, and given actual 2009 innings they should have had a group ERA of 3.48.  Their actual ERA? 3.61.  If anything, not enough regression.

Sean:  It looked to me like in 2007 and 2008 the soft-tossers underperformed, while the hard throwers broke even—making me think the aggregate ERA prediction might have been off in those years.  If not, my suggestion should of course be disregarded. You raise a good point about identifying the worst pitchers.  Maybe the thing to do is to look only at established pitchers (at least 500 career IP, at least 150 IP prior season), and then try to create groups with projected ERAs similar to what John has here.  Something like 1) projected ERA below 4.10; and 2) projected ERA… Read more »

Sean:  OK, sounds like under-regressing the good pitchers isn’t a problem.  But it also sounds like the hard-throwers don’t actually overperform (3.79 vs 3.85 according to your data), so we don’t really need to explain them anyway.

So question is why soft tossers underperform, and whether it’s really velocity or a recent DROP in velocity (or both) that predicts underperformance.

Sean Smith
Sean Smith

Good point.  I’d hate to have been underprojecting Greg Maddux or Jamie Moyer all these years just because they don’t throw hard.  The change might be what to focus on.  I wonder how this looks for relievers too, that’s where you’ll find the best average fastballs.  And again, I’d like to know how these pitchers beat their projections at the component level – strikeout rate is the obvious culprit but who knows.

So many questions I need to hire a staff of sabermetricians!  Too bad I have no money.

J. Cross
J. Cross
(I also posted this at the book blog but thougth I should put it here.) Good stuff, JRM! Not to give it away but Steamer is going to try to use velocity in its 2010 pitcher projections. For the pool of 873 pitchers we looked at the correlation between Marcel’s K/AB and actual K/AB was 0.716.  But the correlation between Marcel “adjusted for fastball velocity” K/AB and actual K/AB was 0.741. In other words, we go from explaining 51% of the variance to 55% of the variance. Disclaimer: This was done by adjusting a Marcel projection for year Y with… Read more »

I see two potential steps to understanding what it is that John has uncovered.

1.  Age-control the sample.  Given that CHONE or PECOTA start with previous years stats, John’s findings seem to indicate a perennial decline by soft-tossers and that begs the question of whether the soft-tossing sample is stacked with older declining pitchers.

2.  Isolate the projected peripherals from ERA to determine whether the increased ERA is coming from year-to-year projections or is hiding within the static peripherals-to-ERA formulas.

Great stuff! One (important) thing:  Please, please, pretty please, with sugar on it, stop with the “keep score method” to analyze these things.  With very few exceptions, that is a BAD (really bad) way to do the analysis. The proper way is to look at the difference between ERA and performance for ALL PLAYERS IN EACH SAMPLE COMBINED, or to do a regression! If you had 10 slow fastball pitchers, and for 6 of them their performance was better than their projection by an average of .25 runs and for 4 of them, their performance was worse that their projection… Read more »
That being said, without weighting by inning, the slow tossers in 08, performed an average of .47 runs worse than their projections and the hard tossers performed exactly at their projections. I did not include those who did not pitch in the majors in 2009.  I am not sure why you put them in the “projection better” category.  I don’t think that is fair to your study or analysis. Plus, I think you have some mistakes.  Pedro is classified as “even” in 2009 despite a .82 run difference and Mark Hendrickson is classified as favoring “performance” despite pitching .17 worse… Read more »

Darrell Rasner pitched in Japan in 2009, where he had a bad season… ERA over 6.00 iirc.



Thanks for the catch. Patrick’s right; the chart’s wrong re: Rasner. Oops. At least he had the courtesy to be bad, supporting the hypothesis.


My guess is simply that the means for the two groups are different and thus Chone’s regressions should be toward different means but are not.  I, for one, do not use velocity as a variable in my projection algorithm, even though I should. I’m pretty sure that John would have come up with the same results if he had used my projections. It would be the same thing if you did not regress hitter stats toward those of players with similar heights and weights, which many projection systems do not do.  If you don’t, you will find that taller players… Read more »

What about comparing projected ERA with luck and defense-neutralized pitching ERA-equivalent stats like xFIP or tERA? I mean, someone may have outperformed their projection because they were lucky, or because a ballpark played differently one year from the next (it happens).  If a player outperformed their projection in ERA, but had an xFIP closer to their projection, that might be a case where we could call that one even…

Rudy Gamble
Rudy Gamble
First off, John.  Interesting stuff. I am a firm believer that projected pitching stats can be improved based on factoring in pitching stats as found on sites like But while I think that pitching velocity has a positive correlation with general performance, I’m not seeing why this would impact projection accuracy assuming the projections are based on fundamentals (K rate, xFIP) vs. performance.  Washburn’s 2009 performance isn’t a vindication that CHONE was a better predictor of performance.  His ERA was an anomaly based on a ridiculously low BABIP (.257) even taking Seattle’s great D into account (F-Her came in… Read more »
Rudy Gamble
Rudy Gamble

One more thought.  Since low velocity throwers tend to have higher ERAs, is it possible that what we’re seeing is that it is harder to project ERA for borderline pitchers vs. good pitchers.  Perhaps it’s a lot more likely that a projected 5.00 ERA pitcher throws for a 4.00 ERA/6.00 ERA than a 3.00 ERA pitches a 2.00 ERA/4.00 ERA….