Comparing 2011 Pitcher Forecasts

This article is the second of a two part series evaluating 2011 baseball player forecasts. The first looks at hitters and found that forecast averages outperform any particular forecasting system. For pitchers, it appears as though the results are somewhat reversed. Structural forecasts that are computed using “deep” statistics (k/9, hr/fb%, etc.) seem to have done particularly well.

As with the other article, I will look at two main bases of comparison: Root Mean Squared Error both with and without bias. Bias is important to consider because it is easily removed from a forecast and it can mask an otherwise good forecasting approach. For example Fangraphs Fan hitter projections are often quite biased, but are very good at predicting numbers when this bias is removed.

Results-RMSE:

system w rank era rank whip rank so rank AVG
Baseball Dope 4.03 3 1.12 4 0.16 1 39.04 1 2.25
Steamer 3.97 2 1.13 5 0.17 7 40.08 2 4
CAIRO 4.29 5 1.11 1 0.16 2 45.03 11 4.75
RotoChamp 4.09 4 1.11 2 0.17 11 40.96 3 5
Razzball 3.97 1 1.12 3 0.17 9 44.06 8 5.25
Marcel 4.71 9 1.14 10 0.16 3 43.58 4 6.5
Fangraphs Fans 4.50 7 1.13 6 0.17 6 45.00 10 7.25
AggPro-System 4.67 8 1.14 9 0.17 5 44.19 9 7.75
Will Larson 4.35 6 1.14 8 0.17 12 43.84 6 8
AggPro-Category 4.81 11 1.13 7 0.17 8 43.93 7 8.25
ESPN 4.78 10 1.16 12 0.17 10 43.76 5 9.25
CBS Sportsline 5.19 12 1.14 11 0.17 4 46.55 12 9.75

Baseball Dope’s forecasts outperform all others based on root mean squared error (RMSE), as is shown in Table 1. Steamer, CAIRO, Rotochamp, and Razzball round out the top 5. Contrary to the hitter analysis, the averaging systems (AggPro and my forecasts) performed relatively poorly. However, because Baseball Dope and Steamer were relatively unknown heading into 2011, each of these averaging systems did not include the Baseball Dope and Steamer projections.

Results-Fit:

system w rank era rank whip rank so rank AVG
Baseball Dope 0.20 3 0.17 3 0.28 1 0.40 1 2
Razzball 0.27 1 0.21 1 0.23 3 0.36 8 3.25
Steamer 0.24 2 0.16 7 0.20 8 0.39 2 4.75
RotoChamp 0.20 4 0.17 4 0.18 11 0.39 3 5.5
CAIRO 0.14 8 0.20 2 0.22 4 0.25 12 6.5
Fangraphs Fans 0.16 6 0.16 6 0.22 6 0.33 9 6.75
AggPro-System 0.14 7 0.16 9 0.22 5 0.36 7 7
AggPro-Category 0.12 10 0.16 5 0.20 9 0.37 5 7.25
Will Larson 0.18 5 0.16 8 0.18 12 0.37 6 7.75
CBS Sportsline 0.07 11 0.15 10 0.23 2 0.28 11 8.5
ESPN 0.13 9 0.13 12 0.18 10 0.37 4 8.75
Marcel 0.02 12 0.14 11 0.21 7 0.30 10 10

This table is the r^2 of the simple regression: actual=b(1)+b(2)*forecast+e.  The b(1) term captures ex-post bias, allowing b(2) to better capture the information content in the forecast.  After correcting for ex-post bias, Baseball Dope ran away with overall title of the best pitcher forecasts of 2011. Razzball projections, after finishing near the bottom in the hitter analysis, finished in second place. Bias correction ends up sending the Marcel forecasts to the bottom of the rankings, as was the case in the analysis of hitters. Clearly, Marcel forecasts do well at producing unbiased forecasts, but do pretty poorly at projection player-to-player variation. Besides Marcel, the forecast rankings do not change very much when looking at overall fit versus RMSE. This implies that each forecasting system has similar bias, and that this bais is greater than what is present in the Marcel projections.

Takeaway:

Baseball Dope was the best at forecasting pitchers in 2011, but since they were new on the forecasting scene, they were not included in the forecast averaging approaches. I would continue to average forecasts among those at the top of the ranking. My guess is that in 2012, forecast averages that contain Steamer and Baseball Dope projections will perform quite well. Time will tell!

Happy forecasting!

All of the non-proprietary numbers in this analysis, as well as my 2012 forecasts, can be found at my little data repository website found at http://www.bbprojectionproject.com.




Print This Post



5 Responses to “Comparing 2011 Pitcher Forecasts”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. brian fawcett says:

    I gather you didn’t include PECOTA because it was more inaccurate than any of the others…

    Vote -1 Vote +1

  2. Will Larson says:

    @Brian: I would have, but they charge. I only consider freely available projections.

    Vote -1 Vote +1

  3. Joe says:

    It doesn’t look like there are any Baseball Dope projections for 2012. The last update on their site is March of last year. Guess they figured they quit while on top.

    Vote -1 Vote +1

  4. zach says:

    did you compare it at all to just the previous seasons stats, too see how much better they are

    Vote -1 Vote +1

  5. Dragon Ash says:

    And the world-wide leader near the bottom. Whoda thunk it.

    Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current ye@r *