Evaluating 2013 Projections

Welcome to the 3rd annual forecast competition, where each forecaster who submits projections to bbprojectionproject.com is evaluated based on RMSE and model R^2 relative to actuals (see last year’s results here).  Categories evaluated for hitters are: AVG, Runs, HR, RBI, and SB, and for pitchers are: Wins, ERA, WHIP, and Strikeouts. RMSE is a popular metric to evaluate forecast accuracy, but I actually prefer R^2.  This metric removes average bias (see here) and effectively evaluates forecasted player-by-player variation, making it more useful when attempting to rank players (i.e. for fantasy baseball purposes).

Here are the winners for 2014 for R^2 (more detailed tables are below):

Place
Forecast System
Hitters
Pitchers
Average
1st
Dan Rosenheck
2.80
2.50
2.65
2nd
Steamer
1.60
6.00
3.80
3rd
FanGraphs Fans
5.80
2.75
4.28
4th
Will Larson
6.60
3.00
4.80
5th
AggPro
6.40
4.25
5.33
6th
CBS Sportsline
5.40
8.00
6.70
7th
ESPN
6.60
7.50
7.05
8th
John Grenci
8.00
8.00
9th
ZiPS
9.80
7.25
8.53
10th
Razzball
6.80
10.25
8.53
11th
Rotochamp
8.60
9.00
8.80
12th
Sports Illustrated
8.80
12.00
10.40
13th
Guru
10.60
12.00
11.30
14th
Marcel
11.20
12.50
11.85

 

And here are the winners for the RMSE portion of the competition:

Place
Forecast System
Hitters
Pitchers
Average
1st
Dan Rosenheck
2.60
2.00
2.30
2nd
Will Larson
3.60
2.50
3.05
3rd
Steamer
1.80
5.00
3.40
4th
AggPro
4.00
3.00
3.50
5th
ZIPS
6.00
5.75
5.88
6th
Guru
4.80
7.25
6.03
7th
Marcel
6.20
8.50
7.35
8th
John Grenci
7.50
7.50
9th
Rotochamp
9.40
9.00
9.20
10th
ESPN
9.20
10.50
9.85
11th
Fangraphs Fans
11.80
8.75
10.28
12th
Razzball
9.40
11.25
10.33
13th
Sports Illustrated
10.60
11.75
11.18
14th
CBS Sportsline
11.60
12.25
11.93

 

I’m beginning to notice some trends in the results across years.  First, systems that include averaging do particularly well.  This is pretty well established by now, but it’s always useful to reflect upon.  It’s been asked in the past to perform evaluations separating forecasts computed by averaging with those that do not include information from others’ forecasts (more “structural” forecasts). I decided not to do this because the nature of the baseball forecasting “season” makes it impossible to be sure forecasts are created without taking into account information from others’ forecasts. This can include direct influence (forecasting as a weighted average of others’ forecasts), but can also occur in more subtle ways, such as model selection based on forecasts that others have put forward.  Second, FanGraphs Fans are always fascinating to me, and how they can be so biased, but yet contain some of the best unique and relevant information for forecasting player variation. The takeaway from the Fans forecast set is that crowdsourced-averaging works, as long as you can remove the bias in some way, or ignore it by instead focusing on ordinal ranks.

Some additional notes: it would be interesting to decompose these aggregate stats in to rates multiplied by playing time, but it’s difficult to gather all of this for each projection system. Therefore, I focus on top-line output metrics.  Also, absolute rankings are presented, but many of these are likely statistically indistinguishable from each other.  If someone wants to run Diebold-Mariano tests, you can download the data used in this comparison from bbprojectionproject.com

Thanks for reading, and please submit your projections for next year! Also, as always, I welcome any comments, and I’ll do my best to respond.

R^2 Detailed Tables

system
r
rank
hr
rank
rbi
rank
avg
rank
sb
rank
AVG
AggPro
0.250
6
0.42
9
0.308
8
0.32
1
0.538
8
6.4
Dan Rosenheck
0.296
3
0.45
1
0.340
3
0.3
3
0.568
4
2.8
Steamer
0.376
1
0.45
2
0.393
1
0.31
2
0.572
2
1.6
Will Larson
0.336
2
0.43
6
0.345
2
0.21
13
0.509
10
6.6
Marcel
0.146
12
0.36
12
0.236
12
0.27
8
0.477
12
11.2
ZIPS
0.118
13
0.42
8
0.230
13
0.3
4
0.504
11
9.8
CBS Sportsline
0.278
4
0.44
3
0.320
4
0.25
10
0.542
6
5.4
ESPN
0.241
7
0.43
5
0.317
5
0.29
7
0.532
9
6.6
Razzball
0.239
8
0.43
4
0.314
6
0.24
11
0.553
5
6.8
Rotochamp
0.234
9
0.41
10
0.287
9
0.23
12
0.569
3
8.6
Fangraphs Fans
0.268
5
0.42
7
0.272
10
0.3
6
0.574
1
5.8
Guru
0.186
11
0.33
13
0.263
11
0.3
5
0.476
13
10.6
Sports Illustrated
0.221
10
0.4
11
0.314
7
0.27
9
0.541
7
8.8

 

system
W
rank
ERA
rank
WHIP
rank
SO
rank
AVG rank
AggPro
0.13
3
0.15
4
0.25
4
0.402
6
4.25
Dan Rosenheck
0.17
1
0.19
2
0.27
2
0.406
5
2.5
Steamer
0.09
6
0.15
3
0.26
3
0.341
12
6
Will Larson
0.16
2
0.19
1
0.24
5
0.413
4
3
Marcel
0.05
14
0.02
13
0.17
9
0.293
14
12.5
ZIPS
0.09
7
0.07
9
0.21
6
0.375
7
7.25
CBS Sportsline
0.1
5
0.08
7
0.15
10
0.359
10
8
ESPN
0.08
10
0.05
11
0.2
7
0.43
2
7.5
Razzball
0.06
13
0.07
8
0.14
12
0.374
8
10.3
Rotochamp
0.08
9
0.06
10
0.17
8
0.359
9
9
Fangraphs Fans
0.11
4
0.08
5
0.28
1
0.435
1
2.75
Guru
0.07
11
0.05
12
0.11
14
0.343
11
12
Sports Illustrated
0.09
8
0.02
14
0.14
13
0.338
13
12
John Grenci

0.07

12

0.08

6

0.15

11

0.42

3

8

 

RMSE Detailed Tables

system
r
rank
hr
rank
rbi
rank
avg
rank
sb
rank
AVG
AggPro
22.495
4
7.34
4
23.217
4
0.03
4
7.096
4
4
Dan Rosenheck
20.792
3
6.91
1
21.867
2
0.03
5
6.467
2
2.6
Steamer
20.355
2
7.02
2
21.817
1
0.03
3
6.258
1
1.8
Will Larson
20.091
1
7.2
3
22.234
3
0.03
8
6.864
3
3.6
Marcel
23.473
6
7.51
6
23.831
6
0.03
7
7.334
6
6.2
ZIPS
25.380
7
7.43
5
25.662
7
0.03
1
8.048
10
6
CBS Sportsline
25.866
10
8.63
13
26.837
10
0.03
12
8.527
13
11.6
ESPN
25.698
8
8.37
12
26.418
9
0.03
6
8.120
11
9.2
Razzball
25.831
9
8.01
9
27.842
12
0.03
9
7.920
8
9.4
Rotochamp
26.199
11
8
8
25.995
8
0.04
13
7.686
7
9.4
Fangraphs Fans
26.854
13
8.12
10
30.804
13
0.03
11
8.289
12
11.8
Guru
23.187
5
7.58
7
23.608
5
0.03
2
7.198
5
4.8
Sports Illustrated
26.609
12
8.24
11
27.173
11
0.03
10
8.009
9
10.6

 

system
W
rank
ERA
rank
WHIP
rank
SO
rank
AVG rank
AggPro
4.4
3
1.031
4
0.17
4
47.01
1
3
Dan Rosenheck
4.25
1
1.014
1
0.17
1
47.9
5
2
Steamer
5.02
8
1.030
3
0.17
2
49.45
7
5
Will Larson
4.34
2
1.017
2
0.17
3
47.44
3
2.5
Marcel
4.62
5
1.158
13
0.18
8
50.84
8
8.5
ZIPS
4.78
7
1.101
7
0.17
5
47.85
4
5.75
CBS Sportsline
5.56
13
1.134
11
0.19
11
57.14
14
12.3
ESPN
5.81
14
1.126
10
0.18
7
53.54
11
10.5
Razzball
5.39
12
1.115
8
0.19
12
55.55
13
11.3
Rotochamp
4.71
6
1.138
12
0.18
9
51.81
9
9
Fangraphs Fans
5.29
10
1.123
9
0.17
6
52.57
10
8.75
Guru
4.51
4
1.093
6
0.19
13
48.79
6
7.25
Sports Illustrated
5.33
11
1.176
14
0.18
10
55.32
12
11.8
John Grenci
5.14
9
1.080
5
0.19
14
47.26
2
7.5

 




Print This Post

7 Responses to “Evaluating 2013 Projections”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. jerusalem-artichoke says:

    thanks for doing this Will yet again it’s one of the most interesting articles on fangraphs all year

    Vote -1 Vote +1

  2. sam says:

    which of these projection systems will be publicly available? seems like the highest performing ones, Rosenheck, AggPro, Larson, are not available, at least not yet

    Vote -1 Vote +1

  3. gabriel syme says:

    So here’s my question: we know that Fangraphs Fan projections have some systematic biases – at least as of a few years ago, they were too optimistic about performance for almost all players. There may be other systematic biases. What happens to the Fangraphs Fan projections if you correct for the systematic biases?

    Vote -1 Vote +1

  4. Will Larson says:

    No problem! In the next several weeks, I’ll begin to put up this year’s projections. It’s a bit early in the season to gather all of them–we still don’t know about all of the pre-season trades/injuries/signings yet. I know Dan Rosenheck’s won’t be publicly available until after the season, but I’ll publish mine.

    In terms of the Fan projections, the R^2 ranking shows how good they are when the bias is removed. I haven’t looked at the relative magnitude of the bias over the last few years, so I’d hesitate to guess how much the bias will be this year. I would suggest that the ordering of the players is probably pretty good, and better than most other publicly available sources out there.

    Vote -1 Vote +1

  5. William Wallace says:

    Should we infer that FanGraph Fans projections are doing a better job of projecting IP for pitchers and thus are pretty good at projecting Ks? Or am I missing something?

    Pitcher rate stats look great for Steamer, but Ks look terrible, so IP projection problems jumped to mind for me.

    Vote -1 Vote +1

  6. He’s no ape, Marcel clearly loves RMSE!

    Hey, nice analysis Will. I was wondering if there is any easy way for you to test out how well, say, if we combined all the projection systems together, how that composite projection would do with your analysis. Or, say, if you combined the top three public forecasts, your forecast plus Steamer plus ZIPS, how that would have done in the rankings. I assume they would be better, just curious how they would have done.

    I was surprised that Oliver was not in the study since their data is in Fangraphs. Why wasn’t it?

    Vote -1 Vote +1

  7. Phil says:

    Any feel for how PECOTA might stack up?

    Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current day month ye@r *