2010 Forecast Evaluations (Part I)

How accurate were our Steamer Projections?  Which system or combination of systems should we use to prepare for our fantasy baseball draft?  Thanks to MGL’s work we already know that Steamer had great success projecting pitcher quality in 2010 and less success projecting batter quality.  Here will we attempt to discover why some systems are having more success than others by breaking hitting and pitching performances into components.  This should also serve as a follow-up to our analysis from a year ago.

Full Disclosure

As the creators of Steamer Projections you may be justifiably skeptical of our ability to serve as an unbiased judge of projection systems.  In an attempt to allay your concerns we are making our data set available here so that anyone can check our data or pursue their own analysis.

The Systems

We examined Steamer and ten of her peers.  Systems were chosen largely based on the extent to which player names were matched with MLBAM IDs.

  • CAIRO – The brain child of SG of Replacement Level Yankee Weblog
  • Chone – Rally’s highly successful but no longer available system.
  • Fantistics – Insider Baseball (Fantasy Sports Site #1)
  • FEIN – Fein Sports (Fantasy Sports Site #2)
  • Guru – Roto Guru (Fantasy Sports Site #3)
  • Marcel – Arguably, the replacement level projection system.  Not because it isn’t good but rather because both the projections and the system are publicly available.
  • MGL – Mitchel Lichtman’s projections
  • Oliver – We used “raw” Oliver projections but playing time adjusted forecasts are available at Hardball Times.
  • Pecota – We used the Player Forecast Manager for projected plate appearances and, for players not listed in the Forecast Manager, rate stats were taken from the weighted mean spreadsheet.
  • Steamer – The high school research project of Dash Davidson and Peter Rosenbloom who, now freshmen in college, are still plugging away.
  • ZiPS – Now seen on ESPN.com and integrated with Diamond Mind Baseball.

The Player Pool

619 batters (non-pitchers) were given one or more plate appearance in 2010.  Since not all systems projected every player, systems that lacked a projection for a given player were assigned Marcel’s rate stats for that player.  They were also assumed to project that player to receive zero plate appearances.

Projected Means

System players PA woBA BB/500 PA K/500 PA BA
Actual 619 289 0.325 43.4 89.8 0.261
CAIRO 592 433 0.327 45.1 83.8 0.259
Chone 610 435 0.332 42.8 88.6 0.267
Fantistics 529 331 0.339 46.1 93.8 0.271
FEIN 502 306 0.341 45.8 85.9 0.271
Guru 506 268 0.326 40.3 87.4 0.264
Marcel 619 298 0.339 45.2 86.9 0.270
MGL 608 298 0.331 45.4* 89.6 0.261
Oliver 615 467 0.326 43.0 88.7 0.262
Pecota 614 298 0.332 45.5 85.7 0.266
Steamer 600 322 0.332 44.8 82.4 0.264
ZiPS 594 442 0.332 43.3 87.9 0.268
System HR/500PA XBH/H R/500 PA RBI/500 PA SB/500 PA
Actual 12.8 0.332 58.6 55.9 6.80
CAIRO 13.7 0.350 62.7 58.9 9.60
Chone 13.9 0.341 62.5 58.9 8.00
Fantistics 15.2 0.342 64.7 61.0 9.10
FEIN 15.1 0.350 68.8 60.1 8.10
Guru 13.6 0.326 62.7 59.2 9.10
Marcel 14.5 0.348 62.5 59.2 8.80
MGL 13.4 0.338 n/a n/a n/a
Oliver 13.1 0.335 58.6 57.4 6.90
Pecota 14.2 0.330 60.8 58.1 8.10
Steamer 14.0 0.356 60.2 58.8 9.10
ZiPS 13.5 0.339 62.6 61.1 7.80

*MGL projects BB and HBP together and I assumed a 10:1 BB:HBP ratio.  For systems that did not project PA it was calculated as (H+BB+HBP)/OBP.

Of the eleven systems, Marcel, Oliver, Pecota and Chone were the most comprehensive systems with projections for nearly all of the 619 players.  These averages for each system don’t tell us which system will help you win your fantasy league but Oliver’s success in predicting league means is a testament to its solid methodology.

Fantistics, FEIN, Guru, Pecota and Steamer appear to be the systems most serious about projecting playing time whereas other systems focus on rate stats and project significant numbers of plate appearance for many borderline players.  Both Marcel and MGL used Community Forecasts for playing time.

Projecting Playing Time (Plate Appearances)

System RMSE
Community 139
Pecota 154
Steamer 156
Guru 160
Fantistics 168
FEIN 168
Chone 233
CAIRO 237
ZiPS 252
Oliver 261

Wow!  The community did a terrific job projecting playing time.  RMSE stands for root mean square error and should givesa sense of how far off the mark a typical projection was for each system.  This is a big win for wisdom of the crowds.  Of the experts, Pecota takes first prize.  The projection systems split pretty neatly into those that try to project playing time and those that don’t.  Maybe everyone should simply join the crowd and use the community’s projected playing time.

Next up…. Projecting Hitting Goodness




Print This Post



9 Responses to “2010 Forecast Evaluations (Part I)”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. Playgu says:

    Very cool. Looking forward to the rest of the evaluation results – just in time for fantasy drafts!

    Vote -1 Vote +1

  2. Oliver produces a playing time forecast which does not distinguish between MLB or minors. However, THT staff and advisers create depth charts which likely do a better job of predicting MLB playing time, and it is the depth charts which are used in the Fantasy Calculator. Either set can be downloaded as a csv file.

    Vote -1 Vote +1

  3. J. Cross says:

    Yes, sorry Brian, I only had “raw” Oliver at my disposal when doing this and I tried to make that clear above. When I gather projections immediately before the season this year I’ll make sure to grab the Hardball Times version with projected playing time.

    Vote -1 Vote +1

  4. evo34 says:

    Good work. Would also love to see a couple other playing time projections evaluated, such as those from Rotoworld and Rotowire.

    Is there any way to combine this year’s analysis with your past year’s work to create a comprehensive evaluation?

    Vote -1 Vote +1

  5. Rudy Gamble says:

    Interesting stuff, J!

    Question: Playing time estimates are hugely important for fantasy baseball. Two points:

    1) Community Projections – I like the fan projections and they seem to do a good job in tests like this and when Tangotiger does them. But they suffer from two shortcomings: a) they are relatively shallow (tough to use for deeper leagues) and b) they are late. It would be interesting to see playing time accuracy comparisons based on time-stamped results (e.g., March 1st, March 15th, Opening Day). While Opening Day would be the most accurate, they don’t help in fantasy drafts. I consider 3/15 to be the most important date.

    2) Interesting to see Pecota over Fantistics as Fantistics beat Pecota in Tango’s 2009 contest….it’d be interesting to run that same test on 2009 info (Tango must have it) to see if BP/Pecota is getting better or there was some regression on Fantistics side.

    Thx,
    Rudy

    Vote -1 Vote +1

  6. J. Cross says:

    evo, I might try to combine multiple years of analysis when I have some time.

    Rudy, that makes sense to me. Unfortunately, I’ve already missed the march 15th date this year but maybe I’ll check around and see if anyone has them.

    Vote -1 Vote +1

  7. Rudy Gamble says:

    Sounds good. You could just use today vs. March 15th. I can get you Fantistic and Rotochamp if you need….

    Vote -1 Vote +1

  8. J. Cross says:

    That would be great. I’ll download the usual suspects tonight and stash these away for next year.

    Vote -1 Vote +1

  9. Joby Leahy says:

    I’d love to see an early look towards next year, it would have an impact as to who I buy my projections from.

    Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>