Adam Dunn hitting 50 homeruns in Comerica would be a feat indeed since he will only play 9 games there this season.
Comment by cobradc23 — February 9, 2012 @ 10:24 am
Matt, one thing that would be interesting but very difficult would be to adjust each of these projections to match the same total run environment. Or alternatively to look at something different see if you can test their predictions on run environment.
I think its very hard to interpret the role we should place on several of these tests in assessing our confidence in various systems projections of individual players. Correlation i guess is important because at the end of the day you want to be ranking players, for fantasy. But when a system makes a call that’s pretty surprising relative to others, thats where I really want to know what’s going on. Any ideas?
Projections?? I scoff at your projections! Evidence? Zoilio Versalles, Darin Erstat, Chris Hoiles, Darren Daulton, Willie Magee, Sixto Lezcano, Darrell Porter, Bob Cerv……The list is huge. That’s why we love baseball so much.
Comment by Hurtlockertwo — February 9, 2012 @ 10:35 am
also, really great work. Steamer really nailed it. I wonder if there isn’t some means of coming up with a similar big find for batters. The first that jumps to mind is, see how they respond to pitcher velocity? That would be an interesting investigation in and of itself.
You’re right– I normalized all projection systems’ weighted average of wOBAs to .330 and ERAs to 3.82. Otherwise, predicting run environment might be more important than predicting players!
Comment by Matt Swartz — February 9, 2012 @ 10:50 am
How did they compare to 3 year Averages?
Comment by Urban Shocker — February 9, 2012 @ 11:05 am
Wow, this article had some really atrocious writing.
“While frequently accused of being overly pessimistic about whoever your Home Nine are, on average, they land high about as often as they land low.”
Who is accused? Who are “they”?
“only including guys with at least 200 PA” actual, predicted? What year?
Also, I’d like to point out for the 100th time that FIP is not an “ERA predictor,” it is an actual measure of fielding independent defense.
How many pitchers were included in your sample? I ran a similar test of ZiPS and one other projection system for 2011 which included 301 pitchers with a similar innings constraint. I got different results for ZiPS, but I take it your sample probably had less pitchers because they had to have been included in all the projection systems forecasts? Let me know, thanks!
Oh interesting. How does that affect “last year”? Did you adjust it, or does this mean that major swings in run environment year to year will affect rmse of “last year”? And why did you pick those points?
It’d be interesting to know what the “last year” stats were going back, because that’d be a good first order way to see if the models as a whole are getting better.
ERA and FIP are operational measures of the same thing–pitcher performance. They take different stands on how to do that, but it is valid to see how they do at predicting one another.
Comment by Barkey Walker — February 9, 2012 @ 11:39 am
I adjusted last year to .330 & 3.82 as well, just by adding/subtracting. I picked .330 because that was something Tom Tango suggested when he gave me the basic wOBA formula he’d suggest using (which gave me something like .325 or so), and 3.82 because that was the league average ERA last year when exclude pitchers below 40 IP.
Comment by Matt Swartz — February 9, 2012 @ 11:45 am
Sorry if it wasn’t clear:
they = projections
200 PA = 200 PA in 2011 in real life
Also, I called FIP an ERA Estimator, not an ERA Predictor. I try not to call things ERA Predictors unless they are predictions.
Comment by Matt Swartz — February 9, 2012 @ 11:48 am
Getting to practicalities: if I’m in a very deep, NL-only, auction keeper league where you win predominantly based on buying low at auction on emerging talent that you can then lock up for years, I should use … what system for batters? Pitchers?
I think it was 355 or 359 pitchers or something like that. It was everyone with 40 IP, and I filled in numbers for “blank” projections by doing a slightly below average wOBA for hitters (I think .310) and a slightly above average ERA for pitchers (I think 4.12, but I don’t remember), with the exception of Marcel, where I was told it’s supposed to be league averages. There weren’t many missing projections, though, IIRC.
Comment by Matt Swartz — February 9, 2012 @ 11:50 am
I think if you averaged the best projection systems for batters, that does a lot better IIRC. A long time ago, I checked projections on subsets of batters and found major differences. I think ZiPS used to be very good with older players and PECOTA used to be very good with very young players, but I think both systems have changed a lot since then. Maybe I’ll check that one day (or you can do it first and use it in your keeper league before I publish anything!).
Comment by Matt Swartz — February 9, 2012 @ 11:52 am
This is the year he will hit 50 in Comerica! And about 30 in the other 150 games.
Just had it pointed out that the way that I fitted SIERA_proj, xFIP_proj, FIP_proj, and tERA_proj, I’m going to get positively biased results since I used 2011 data to test 2011 data. I’ll redo this analysis in the next few days and make note of it here.
Comment by Matt Swartz — February 9, 2012 @ 12:39 pm
any idea if/when the 2012 version of steamer will be available? or marcel here on fangraphs for that matter? i know ZIPS is rolled out one team at a time before dan releases the entire series and that he’s almost done.
Comment by johnnycuff — February 9, 2012 @ 12:59 pm
Good question. First thing I did was go looking for the Steamer download.
I was really hoping someone would do something like this.
The Steamer pitcher information strikes me as particularly heartening. I’m glad to hear my views on the subject are put to good use (my article is linked toward the end, and I’m calling one of my fantasy teams the Staggering Geniuses this year; if you hit both links there, you’ll get that.)
I did talk talk to a few projections-related people about velocity-based projections (and I’ve been tinkering with my own pitcher projections for a bit.) Some dismissed the idea outright or close to it; Sean Smith (formerly of CHONE fame) explained to me prior to the article why he thought it wouldn’t help (and was exceedingly helpful and gracious when I explained why I thought it would.)
Use of handedness is a critical piece of knowledge when using velocity. Much less critical, but still informative, is the use of percentage of fastballs.
On an aside, I’m surprised PECOTA did as well as it did. They had some known breakages in their projections (the Kila-Bowker problem, and minor leaguers generally.) As I think about it, Kila and Bowker and other similarly situated folks didn’t meet the 200 PA cutoff – so they didn’t get factored in. So maybe I shouldn’t be as surprised.
Comment by John R. Mayne — February 9, 2012 @ 4:57 pm
Buckets! How did they do in prediction the top 25% of players, etc.?
Would you care to publish the AAE after adjusting for overall offense level? That is, if a system projected a run environment 0.10 runs too high, all of its projections would get reduced by 0.10 before being evaluated. Just trying to separate overall run environment prediction skill from indiv. player differentiation skill. I get that the correlation analysis provides some of the latter, but would like to see a bias-adjusted AAE of possible. THanks.
I don’t think it’s reasonable to fill in numbers for “blank” projections with anything. By definition, these guys got more playing time than expected in 2011 and therefore probably preformed closer to league-average (better) than expected. So you are giving an advantage to systems that didn’t project these players.
A fairer approach would be to simply drop from your analysis those players that did not have projections from every system (excluding Marcel). I hope you will consider re-running the analysis, as I don’t think it fairly judges the systems in its present form.
Is there a way we can measure bat speed? If the velocity is a way to measure pitchers and predict, maybe we can look at bat speed similarly. The notion is that if you have a quick bat, you obviously hit the ball harder, but you also have longer to wait on a pitch and see it.
Comment by Antonio Bananas — February 12, 2012 @ 8:42 am
impressive that using a simple weighted SIERRA estimate whoops nearly all pitching projection systems. Can you publish a spreadsheet for 2012 ERA estimates based on your SIERRA formula?
one thing I’m curious about is if an average of the all systems performs better overall? perhaps by “smoothing” out the errors of the wildest projections by any “outlier” system. Can you add that to the data? (especially for hitters)
this “grading” of projection systems is really a fertile topic that deserves a lot more digging into. Please keep more coming! Which systems perform best at which “types’ of players? old and established? young and with limited major league data? etc.
Unfortunately, I don;t think it’s really a fair test bc the projection systems attempt to predict perf. of nearly everyone who had any chance of having 40 IP in the coming season, whereas the ERA estimators got a free pass on those pitchers with no prior major league data. Why is it a free pass? Because they got to skip the very hardest players to project. And those players who did actually do well enough to get 40 IP the next year, the ERA estimator systems were able to use a league-average projection, essentially *after* the fact.
So unless I am misinterpreting Swartz’s methodology, he really should re-do the analysis after dropping any players not projected in all systems being preprared. I.e., do one comparison with estimators, Marcel and all others (of major league vets only); do one comparison of only projection systems (that includes virtually all players).
I have some of the older projections too, though not for every system. I’ll try to test some of the older systems. I’ll need to look at the link you posted, though I don’t know too much about football data.
I actually included ERAs .20 higher than league average for any missing data, so it didn’t really give anyone a free pass in that case– it actually limited their projection numbers to have them all given the same ERA for those pitchers.
I am one of the creators of Steamer Projections and we are working on the 2012 version of the system as we speak, hoping to have pitcher projections (to begin with, hitters to follow shortly after) by the end of the week. You will be able to find and download the projections here: http://www.steamerprojections.com
I am one of the creators of Steamer Projections and we are working hard trying to get our 2012 projections to you guys as soon as possible. We are hoping that the steamer pitcher projections (with hitters soon to follow) will be out by the end of the week.
I agree with this idea. I would imagine that a significant amount of the error in projections occurs in fringe players. Fantasy players want projections for the top 250 players or so, so it would make sense to judge projection systems based on (mostly) established players.
Even average +0.20 is a free pass. Most of these players were not expected to get playing time in 2011, so the true projection systems would likely project really bad ERAs for them as a group. By allowing the ERA estimators to insert an average+0.20 “projection” after having the knowledge that the player earned significant playing time, is an unfair advantage. The test should be broken into two and should use different universes of players, depending on whether the system is a true projection system or simply an estimator of past performance by MLB veterans.