You Should Trust the Projections

Here on FanGraphs, a lot of the data we present is built around future projections. We host both pre-season and in-season forecasts from multiple systems, and these forecasts form the basis for many of our tools, including our Playoff Odds model. It’s safe to say that we like forecasts.

But I know not everyone reading this post trusts the projection systems, especially when they are forecasting things that we don’t want to be true. It’s natural to want to discard prior information when it doesn’t align with what we want to believe, and often, the forecasts serve as a wet blanket alternative to enthusiasm and excitement. What’s to like about a system that takes the fun out of breakout performances, or tells us that we need to be more patient with the guy who just looks terrible every time we watch him?

And it’s not like projection systems are infallible. Those forecasts never saw Cliff Lee coming, and they certainly didn’t anticipate Jose Bautista was going to go from career scrub to superstar at the age of 29. There are plenty of examples where players made dramatic, unexpected turns in their career arc, and using their track record to project their future would have been wrong. The fact that there are players who have radically changed their performance base allows us to extrapolate that possibility to every player who is performing in a way inconsistent with their projection.

But we just shouldn’t do it. Projections might be fun-sucking, soulless algorithms that are incapable of picking up on adjustments that players can and do make, but the reality is that these forecasts do a pretty good job of predicting the future, even for players whose current performances and projections don’t line up at all.

Mitchel Lichtman has done the heavy lifting here, and has two great posts up at his blog on this topic. I’m going to borrow liberally from the posts because the conclusions are important, but don’t short yourself and just read these excerpts; go read the entirety of both posts. It’s worth your time.

Let’s start with his post on hitters.

I have a database of my own proprietary projections on a month-by-month basis for 2007-2013. So, for example, 2 months into the 2013 season, I have a season-to-date projection for all players. It incorporates their 2009-2012 performance, including AA and AAA, as well as their 2-month performance (again, including the minor leagues) so far in 2013. These projections are park and context neutral.

We can then compare the projections with both their season-to-date performance (also context-neutral) and their rest-of-season performance in order to see whether, for example, a player who is projected at .350 even though he has hit .290 after 2 months will perform any differently in the last 4 months of the season than another player who is also projected at .350 but who has hit .410 after 2 months. We can do the same thing after one month (looking at the next 5 months of performance) or 5 months (looking at the final month performance). The results of this analysis should suggest to us whether we would be better off with Butler for the remainder of the season or with Gomez, or with Hosmer or Morse.

I took all players in 2007-2013 whose projection was at least 40 points less than their actual wOBA after one month into the season. They had to have had at least 50 PA. There were 116 such players, or around 20% of all qualified players. Their collective projected wOBA was .341 and they were hitting .412 after one month with an average of 111 PA per player. For the remainder of the season, in a total of 12,922 PA, or 494 PA per player, they hit .346, or 5 points better than their projection, but 66 points worse than their season-to-date performance.

Here, we see some evidence that perhaps the rest-of-season forecast is underweighting the recent performance relative to past track record, but it’s doing so to the tune of five points of wOBA, which is not much at all. What about guys on the other end of the spectrum? How do the projections work for guys who look totally lost at the plate?

There were 92 such players and they averaged 110 PA during the first month with a .277 wOBA. Their projection after 1 month was .342, slightly higher than the first group. Interestingly, they only averaged 464 PA for the remainder of the season, 30 PA less than the “hot” group, even though they were equivalently projected, suggesting that managers were benching more of the “cold” players or moving them down in the batting order. How did they hit for the remainder of the season? .343, or almost exactly equal to their projection.

The optimistic “don’t worry about it, it’s just a slump” forecast basically nailed the future performance of the guys who looked the worst at the plate. When a guy is 70 points under his wOBA forecast, it’s easy to think that maybe he’s dealing with an injury or some off-the-field issue that the projection system isn’t accounting for, or maybe pitchers are exploiting a flaw that they hadn’t found previously, but in the long-term, these guys just went right back to being good hitters.

But maybe you don’t care about a one month slump or hot streak. Mabe once it gets to two or three months, you don’t consider it randomness anymore, and now you think the forecasts are definitely missing something. Here’s some more data.

About half into the season, around 9% of all qualified (50 PA per month) players were hitting 40 points or less than their projections in an average of 271 PA. Their collective projection was .334 and their actual performance after 3 months and 271 PA was .283. Basically, these guys, despite being supposed league-average full-time players, stunk for 3 solid months. Surely, they would stink, or at least not be up to “par,” for the rest of the season. After all, wOBA at least starts to “stabilize” after almost 300 PA, right?

Well, these guys, just like the “cold” players after one month, hit .335 for the remainder of the season, 1 point better than their projection. So after 1 month or 3 months, their season-to-date performance tells us nothing that our up-to-date projection doesn’t tell us. A player is expected to perform at his projected level regardless of his current season performance after 3 months, at least for the “cold” players. What about the “hot” ones, you know, the ones who may be having a breakout season?

There were also about 9% of all qualified players who were having a “hot” first half. Their collective projection was .339, and their average performance was .391 after 275 PA. How did they hit the remainder of the season? .346, 7 points better than their projection and 45 points worse than their actual performance.

Again, we see some evidence that the forecasts are not putting enough weight on the recent performance, but it’s a half dozen points of wOBA, and the forecasts are much closer to reality than thinking that the current performance is their new level of talent.

But that’s hitters. What about pitchers? Pitchers can add new pitches, tweak their mechanics, move to the other side of the mound, or change arm-angles. They can add and lose velocity. They can learn a new grip on a pitch. There are so many additional variables that forecasting pitchers should be much more difficult, and we’ll need to lean more heavily on current performance rather than trusting the forecasts to the same degree that we trust them for hitters, right?

After one month, there were 256 pitchers, or around 1/3 of all qualified pitchers (at least 50 TBF), who pitched terribly, to the tune of a normalized ERA (NERA) of 5.80 (league average is defined as 4.00). I included all pitchers whose NERA was at least 1/2 run worse than their projection. What was their projection after that poor first month? 4.08. How did they pitch over the next 5 months? 4.10. They faced 531 more batters over the last 5 months of the season.

What about the “hot” pitchers? They were projected after one month at 3.86 and they pitched at 2.56 for that first month. Their performance over the next 5 months was 3.85. So for the “hot” and “cold” pitchers after one month, their updated projection accurately told us what to expect for the remainder of the season and their performance to-date was irrelevant.

MGL’s “NERA” metric is component-based, so you’re not just seeing the vagaries of team defense or ball in play luck evening out over the course of the season. You’re looking at guys who were legitimately pitching well or poorly, the Aaron Harangs and Marco Estradas of the world. And as he notes, there is even less evidence that in-season performance tells us something about pitchers who get off to a one-month start that differs from what the forecasts would have expected.

What about if we break it out by pitcher-type, based on what we know about pitchers already?

In fact, if we look at pitchers who had good projections after one month and divide those into two groups: One that pitches terribly for the first month, and one that pitches brilliantly for the first month, here is what we get:

Good pitchers who were cold for 1 month

First month: 5.38
Projection after that month: 3.79
Performance over the last 5 months: 3.75

Good pitchers who were hot for 1 month

First month: 2.49
Projection after that month: 3.78
Performance over the last 5 months: 3.78

So, and this is critical, one month into the season if you are projected to pitch above average, at, say 3.78, it makes no difference whether you have pitched great or terribly thus far. You are going to pitch at exactly your projection for the remainder of the season!

Yet the cold group faced 587 more batters and the hot group 630. Managers again are putting too much emphasis in those first month’s stats.

What if you are projected after one month as a mediocre pitcher but you have pitched brilliantly or poorly over the first month?

Bad pitchers who were cold for 1 month

First month: 6.24
Projection after that month: 4.39
Performance over the last 5 months: 4.40

Bad pitchers who were hot for 1 month

First month: 3.06
Projection after that month: 4.39
Performance over the last 5 months: 4.47

Same thing. It makes no difference whether a poor or mediocre pitcher had pitched well or poorly over the first month of the season. If you want to know how he is likely to pitch for the remainder of the season, simply look at his projection and ignore the first month. Those stats give you no more useful information.

MGL goes through and repeats the process for larger samples for pitchers as well, and finds the same thing. In-season forecasts predict future performance extremely well. Even though pitchers can make all kinds of tweaks, the projections handle the outliers better than treating them as outliers and assuming the forecasts just missed something. Even at four or five month samples, the forecasts beat season-to-date performance.

This doesn’t mean the forecasts are correct 100% of the time, of course. There are exceptions. Cliff Lee and Jose Bautista do exist. But the existence of exceptions doesn’t mean that we can use seasonal performance to identify the next Cliff Lee or Jose Bautista. If you’re setting out to find the next Lee or Bautista, you have to use something other than seasonal performance in order to identify which guys are “for real” and which guys aren’t, because the seasonal numbers won’t help you find the true exceptions to the rule. If you’re going to assume that a forecast for a player is no longer valid, you need something other than performance-deviating-from-that-forecast to support your argument.

When a true breakout like that happens, the forecasts will be slow to identify it, and will lag months behind until it finally weights the new data heavily and decides “hey, this guy is good now.” But the alternative is reacting far too quickly to seasonal performance, changing your opinions like Tony LaRussa changes relievers. Without perfect information, we’re going to be wrong on some guys. The evidence suggests the conservative path, leaning almost entirely on forecasts and putting little weight on seasonal performance, is the one that is wrong the least.



Print This Post



Dave is the Managing Editor of FanGraphs.


Sort by:   newest | oldest | most voted
jon
Guest
jon
1 year 11 months ago

Interesting. Just a thought: how can we automatically blame the managers for giving the cold-starters less innings and at bats? Maybe some people in the cold-starting group actually WERE dealing with injuries which were helping them get off to a cold start and therefore were more likely to end up on the DL. For example, this year something didn’t seem right about Fielder, and it ends up injuries belied his cold start. Same with others.

MGL
Guest
MGL
1 year 11 months ago

Yes, that is true to some extent. But surely managers ARE giving cold hitters and pitchers less playing time and the hot ones more playing time solely based on their stats. After all, virtually ALL managers believe in hot and cold players (just based on the stats) and NO manager looks at player projections.

Richie
Guest
Richie
1 year 11 months ago

We are sure Maddon isn’t looking at player projections? (just for an example) I wouldn’t expect ’em to ‘fess up to it, for a couple of reasons.

Benefit of the doubt
Guest
Benefit of the doubt
1 year 11 months ago

Some changes in performance are real; some are luck. If you were a manager and couldn’t tell where a large change in performance was real or lucky, which way would you be biased? I would probably give the players the benefit of the doubt since the alternative would mean not rewarding or punishing performance. The players would have no incentive to improve if a manager relied exclusively on projections

Richie
Guest
Richie
1 year 11 months ago

Excellent insight, thank you. (interjecting that players would still have long-run incentives, but we human types respond much more energetically to shorter-term ones)

Tim A
Guest
Tim A
1 year 11 months ago

The projections always hate the A’s, and I think a big part of the reason why is that the projection system doesn’t know how to properly evaluate what the team does with platoons. I think weighting the data toward the stats accumulated since they started batting in the platoon might be relevant.

ctrain03
Member
ctrain03
1 year 11 months ago

Where do I find the NERA stat? I would love this information. Also, what projections are you using?

Joe
Guest
Joe
1 year 11 months ago

These are MGLs proprietary projections. NERA I believe is also a proprietary MGL stat

MGL
Guest
MGL
1 year 11 months ago

Yes, NERA is simply Base Runs (which is a component RA) normalized to 4.00, as 4.00 is league average. It is based on pitcher components (s,d,t,hr,bb,k,hp,WP) whether they be actual or projected.

All of my projections are context neutral and the actual stats are context neutral as well. That is, I factor out (try to at least, it is part science and part art) parks, defense (including catcher framing), opponents, and umpires.

MGL
Guest
MGL
1 year 11 months ago

Nice job Dave! The astounding thing to me is that even after 5 months, with only one month to go in the season, the season-to-date pitcher stats were completely worthless in terms of adding information to the projections. In fact, using those 5 month season-to-date stats as a proxy for a pitcher’s true talent or to project his performance over that last month is a huge mistake (assuming that the projection and the seasonal stats are divergent). Take a look at this data 5 months into the season. I mean REALLY take a look at it.

Poor pitchers who were cold for 5 months

First month: 5.45
Projection after 5 months: 4.41
Performance over the last month: 4.40

Poor pitchers who were hot for 5 months

First month: 3.59
Projection after 5 months: 4.39
Performance over the last month: 4.31

Good pitchers who were cold for 5 months

First month: 4.62
Projection after 5 months: 3.72
Performance over the last month: 3.54

Good pitchers who were hot for 5 months

First month: 2.88
Projection after 5 months: 3.71
Performance over the last month: 3.72

Jaker
Guest
Jaker
1 year 11 months ago

This is exactly why when every FG writer states that X stabilizes after X PA is wrong, particularly since Pizza Cutter’s data that they use as proof doesn’t actually prove that at all.

AK7007
Member
AK7007
1 year 11 months ago

Every? And most of the time I see those types of things talked about, they are in relation to plate discipline or some other component stat for a player new to the league. Not big picture value stats.

MGL
Guest
MGL
1 year 11 months ago

The “stabilization” thing is a different animal. For one thing, when analysts talk about a stabilization point, they mean the number of opportunities at which you would regress 50% toward some mean.

More importantly, that does not mean in one season only, and then ignore past seasons. That is the problem. Many people do that and it is just plain wrong.

John Havok
Guest
John Havok
1 year 11 months ago

Do you have the same type of projections based on how much experience the player has? You have the projections for good pitchers vs bad pitchers, what about Pitchers or hitters in the first say 1-3 years of career (would need an IP or AB threshold for each segment naturally) vs, 4-6 years, 7-10, 10+ etc?

MGL
Guest
MGL
1 year 11 months ago

I’ll maybe split the data up and look at various subsets of players.

Kris
Guest
Kris
1 year 11 months ago

This isn’t really groundbreaking info, is it? Of course in the aggregate over a larger sample size, projections are going to smooth out. It’s the outliers and one-off cases that make baseball fun and worth investigating.

Kris
Guest
Kris
1 year 11 months ago

should edit – didn’t mean that it’s not interesting to read about/see that they does hold true

Jason B
Guest
Jason B
1 year 11 months ago

You should head over to FanGraphs – anytime there are rankings, you’ll see lots of “HOW CAN YOU NOT BELIEVE SO-AND-SO IS THE SECOND COMING OF BABE RUTH, HE HAD AN AMAZING MONTH!” and “THAT DUDE SUCKS HIS ERA WAS OVER 5 OVER HIS LAST FOUR STARTS!!” etc etc. Gobs n’ gobs of recency bias.

Kris
Guest
Kris
1 year 11 months ago

I’m at Fangraphs! Did you mean ESPN comments or something

Brian
Guest
1 year 11 months ago

No no this is fake Fangraphs, we got you! Head over to REAL Fangraphs. It’s over there.

Jason B
Guest
Jason B
1 year 11 months ago

Oops! RotoGraphs. It comes a lot in the fantasy-based discussions and rankings.

MGL
Guest
MGL
1 year 11 months ago

No, I don’t think it is groundbreaking at all. I just wanted to emphasize that all the nonsense you read and hear day in and day out about how so-and-so is X for the season, therefore they or the manager should be doing Y, is just…nonsense… For the most part.

Rally squirrel
Guest
1 year 11 months ago

Fredi Gonzalez excluded.

bluestraggler
Member
Member
1 year 11 months ago

Exactly what I was thinking the whole article. Super interesting to see this confirmed, but yeah, all of this is in aggregate. Individual mileages may vary.

atoms
Guest
atoms
1 year 11 months ago

This is exactly what I came down to comment. The projections work well for large groups, but in fantasy baseball and even in normal baseball, we’re not concerned about a large group of underperformers or overperformers; we are interested in evaluating the guy on OUR TEAM who’s the outlier and determining whether or not he’s going to return to normal or if this is the new normal.

So yeah, as a group, underperformers and overperformers end up normalizing to the projection, but how variable were they as individuals? I bet it’s a significant amount.

Thor
Guest
Thor
1 year 11 months ago

You’re exactly the kind of guy I like to take advantage of in my fantasy league. You think you can predict which of the guys is going to be the outlier (the variable) and either keep him when he’s abnormally hot, or punt him when he doesn’t perform. That’s the whole point of the article, that, yes, there will be some outliers, but the statistics tell us that all that is just noise. The goobs on your fantasy team are much more likely to hit their projections for the rest of the season than they are to continue doing what they are doing. But hey, if you can pick the outliers, then you are really awesome. That’s why your team log for a guy will be .220, 3 HRs, 17 RBIs through the end of May, and I’ll pick that guy up and my team log for him will be .280, 17 HRs and 72 RBIs the rest of the way. I want to thank you, because it’s guys like you that still make fantasy baseball fun for those who understand.

jacaissie
Member
1 year 11 months ago

The “yet the group doing poorly received less playing time” is cited as evidence that managers are looking too hard at the slumps and not using the projections. Isn’t it also potentially evidence that there is some survivor bias in the data? Couldn’t it be that some slumps are actually indicators of injury or a true diminishment of skill, and managers actually stop playing the hurt/ineffective players?

MGL
Guest
MGL
1 year 11 months ago

Yes, it could be. However, don’t forget that managers are the worst offenders of recency bias. And we are only talking about a 5% difference between future playing time of the hot and cold players, so there can’t be TOO much survivorship bias going on. In any case, my projections at least, are based on “if the player does indeed play.”

jacaissie
Member
1 year 11 months ago

I suppose for my purposes, either as a fan or a fantasy player (or a team), I do care about that 5% playing time. If we can predict that a guy is going to be totally useless about 5% of the time after he has significantly underperformed, that is important information. For the fan or fantasy player, it almost doesn’t matter. For the GM, it’s worth knowing the difference between recency bias and actual diminishment in skill. I’m sure there’s a clever way to tease out the actual reduction in skill from the recency bias of managers in that 5% reduction in playing time. Although you’re already working with a relatively small dataset, so this might be tricky.

regression
Guest
regression
1 year 11 months ago

Of course projections are right on average, but it’s not very interesting to be right on average and miss all the outliers who have changed something significant in their performance. Don’t write off Cliff Lee, Corey Kluber, and Dallas Keuchel by dismissing them as unusual. The whole point of baseball analysis is to find the unusual and get an edge on the people who use all the regular methods. Regression to the mean is well understood enough now that it’s not going to win you many games.

If the goal is to be right more often than to be wrong, just always bet on regression to past performance. If the goal is analysis that pushes knowledge forward, we should dig deeper when we notice big changes.

BDF
Guest
1 year 11 months ago

Or if the goal is just to have some fun or some interesting conversations or test your amateur’s scouting eye or indulge in a little wish fulfillment. If the goal is just to be right–and for many folks I suppose that is almost always the goal–or win bets then the “should” of Dave’s title hits the mark. If your goal is something else, it doesn’t.

That having been said, as someone who doesn’t have these conversations in a “being right is the most important thing” context very often, results like these are still incredibly interesting and inform the non-right/fun-type conversations I do have. It’s a rich tapestry.

Daniel
Guest
Daniel
1 year 11 months ago

Many saw Kluber coming

Jason B
Guest
Jason B
1 year 11 months ago

There are weekly meetings for that to be discussed in reverential tones. *Forwards application for Corey Kluber fan club from C. Cistulli, Dictator-for-life*

Fred
Guest
Fred
1 year 11 months ago

Thanks for the article Dave. This reminded me of something I was wondering back at the beginning of the season. Are there specific types of players who have especially large amounts of uncertainty around their projections? Clearly players who are new to the league fit that description, but are there other groups? I think that type of information could be helpful in evaluating team projections. Maybe not though. Maybe on the level of the team projection errors will tend to average out.

Confidence Intervals
Guest
Confidence Intervals
1 year 11 months ago

I agree with Fred 100%. In what ways are projections systems systematically biased? To answer this kind of question, we would need something like a confidence interval around performance, and this could be done pretty easily by steamer or anyone. To not show any type of uncertainty estimates gives the incorrect precision of accuracy where there is actually a lot of error, some of which is systematic.

MK
Guest
MK
1 year 11 months ago

Guys that depend on infrequent events are more susceptible to wide ranges. Examples include guys that depend on home runs and defense for value. Remember one season of data is not stable for projecting defensive metrics.

Confidence Intervals
Guest
Confidence Intervals
1 year 11 months ago

Agreed. And there are simple statistics that could be shown along with the projection to give fans and fantasy baseball users a sense of how certain a projection is. By not including any indication of error, and by writing pseudo-science articles implying that projections are tantamount to omniscience, FG implicitly supports an inaccurate view of projection systems and statistical inference more generally. If you go back to the first public projection system, Marcel, you will find an uncertainty estimate for each player, as I recall. This is because TangoTiger knows that all point estimates are largely meaningless without associated margins of error or confidence intervals.

jfree
Member
jfree
1 year 11 months ago

Thank you for the spinach. Highly nutritious and I already feel healthier and less impulsive

Daniel
Guest
Daniel
1 year 11 months ago

Good stuff, but you’ve just reported means. I’d like to see some boxplots and focus on those outliers to see if there is anything strange about those individuals — e.g. recovering from injury, different batting stance, etc. Also, is there any difference between players that are still “learning”, e.g. in their first 2,3,4 years vs. veterans?

MGL
Guest
MGL
1 year 11 months ago

My next post on my site will be the distribution of future performance for the hot and cold players. My guess? Not much different than for the “consistent” players (those whose seasonal stats are in line with their projections). Maybe a little wider, but overall, not much different.

I have yet to see a study where ANYONE can target a group of players who are indeed going to significantly outperform or underperform their basic projections – IOW, identify those “outliers” who indeed had a change in true talent that is significantly greater than the updated projection suggests. I am not saying that it can’t be done, but I am waiting. Unless I missed something. If I did, someone let me know.

Selection bias
Guest
Selection bias
1 year 11 months ago

Dear MGL,

It seems like there could significant selection bias on the lower end of the performance distribution that would actually favor projection systems. That is, hitters/pitchers who are really bad get demoted/platooned/benched/DL’ed such that they would never make it into your sample at all. More generally, regression to the mean means that a sample of extreme performances at either end of the distribution is always likely to show regression to the mean. This observation about statistics was first made by Stanley and Campbell in 1963. Perhaps breaking up the sample into quintiles and showing performance vs projection for each quintile would help to address regression to the mean.

Wobatus
Guest
Wobatus
1 year 11 months ago

Cool, looking forward to that. Nice work btw.

Eric M. Van
Guest
1 year 11 months ago

I think by definition it can’t be done for a group of players. If there were some generalizable thing that players could do to alter their true talent level, they’d be doing it. And they’re not.

Individual players can be identified, and in my experience you hardly ever follow the same logic and look at the same evidence for two different guys. Each case is essentially unique.

Here’s an example. There was a year where David Ortiz very noticeably altered his stance, by going into somewhat of a crouch rather than being upright. We knew the date this happened, so there were no arbitrary endpoints. He immediately started hitting LHP hugely better, the differences passing tests of statistical significance with ease. From a scouting perspective, there was no mystery; you could see that the new stance had closed a hole he used to have, low and inside. (Nowadays, of course, we could verify that easily with pitch/fx data.) I very confidently asserted that he had actually improved against LHP and that the improvement was no SSS fluke and would be sustained. I was mocked by some (right here, in fact!), but I was right.

This sort of thing is rare, though. That’s an understatement. When you suspect that a player is doing something different, whenever possible you want to make a prediction about some stat or split that you haven’t looked at yet, which would follow if he has really altered his approach in the way you think he has. More often the not, the results are inconclusive.

Nick
Guest
Nick
1 year 11 months ago

You’re gonna need Tommy John surgery from patting yourself on the back so hard.

Cliff
Guest
Cliff
1 year 11 months ago

I can target a player who will indeed significantly outperform his basic projections: Dellin Betances

Wobatus
Guest
Wobatus
1 year 11 months ago

So I guess there is some hope for the Royals (Hosmer and Butler).

szielinski
Member
Member
szielinski
1 year 11 months ago

Any prediction of the future generates risk as a byproduct. how could it be otherwise when a prediction makes factual claims about events which are not facts. The risk produced by a well-constructed projection system is less than the risk produced by eye-first scouting. Of course, the risk generated by projection systems depends upon sample size. Even the eye-test first predictions benefit from larger sample sizes. But what projection systems do very well is to identify factors that can inhibit or undermine player performance: The high strikeout-rate hitter; the pitcher lacking pitch command; etc. This seems to be the junction point between statistical analysis and scouting analysis.

Projection systems perform poorly when identifying future outliers. This failure issues from the nature of statistical analysis. Scouts can identify outliers, but it is not as though they are all blessed with God-like insight into what makes a great baseball player. It’s just that they may be more likely to generate insight into a rare case than a projection system which, by design, fails to handle rate cases.

szielinski
Member
Member
szielinski
1 year 11 months ago

“…fails to handle rate cases.”

This phrase should read: “…fails to handle rare cases.”

Wobatus
Guest
Wobatus
1 year 11 months ago

I’d think some things to look for are, is the break out fueled by something more sustainable. Is it babip fueled, in which case perhaps suspect if not accompanied by improvements in say, contact rates. Has the player started walking more, swinging less at outside pitches, or is there an uptick in a player’s flyball rate? I seem to recall Bautista’s huge September in 2009 somehow presaged what was to come, at least for some observers.

Tom Cranker
Member
Tom Cranker
1 year 11 months ago

Great article. I will keep this in mind except that it doesn’t apply to the young players on my favorite team, who are all undervalued by the biased projection system.

szielinski
Member
Member
szielinski
1 year 11 months ago

…it doesn’t apply to the young players on my favorite team, who are all undervalued by the biased projection system.

Of course it doesn’t apply!

Richie
Guest
Richie
1 year 11 months ago

No, Dave has it in for your favorite team. He’s a hater.

Richie
Guest
Richie
1 year 11 months ago

No, it’s Dave who’s biased against your favorite team. He’s a hater.

Richie
Guest
Richie
1 year 11 months ago

Doggone it, the first one didn’t show up till after I repeated it. Oh well, some jokes are so great you should tell ’em twice.

Stan "The Boy" Taylor
Member
Stan "The Boy" Taylor
1 year 11 months ago

Well, it applies when they are off to a slow start, but not the other way around, obviously…

jvetter
Guest
jvetter
1 year 11 months ago

Excellent read and fascinating results. Thanks.

Brad
Guest
Brad
1 year 11 months ago

So if 5 months of season worth of data doesn’t predict the 6th month of performance better than the projections then how will a whole season worth of data of performance this year be used to predict next year’s performances. I’m a novice here so be kind.

Eric R
Guest
Eric R
1 year 11 months ago

It is that the first x months plus recent years trumps the first x months. What you propose is a further extension of projecting using just the last year versus what would then be the previous four years.

As said above– the take away is the more information is almost always better.

That said, 3 years and a month of data vs a month of data and knowing the pitcher hurt himself and needs TJS favors the latter by quite a bit :)

Tangotiger
Guest
Tangotiger
1 year 11 months ago

MGL *UPDATED* his forecasts to INCLUDE the recent performance.

He simply didn’t give it some absurd overweight.

And his results are showing that the amount of weighting he’s giving the recent performance is a good weight.

I don’t know how much weight he’s giving, but typically, we’d give each year 70 to 80% of the weight of the more recent year.

For those who believe in “recency”, you would therefore want to give a weighting scheme like this:
100% current year
40% previous year
30% year before that
20% year before that

Or some such pattern, to capture the recency-istas belief. And MGL is showing he doesn’t even have to bother with recency beyond the standard scheme.

Xeifrank
Guest
1 year 11 months ago

As you know it is more accurate to weight on a daily basis not a yearly basis. The difference is large enough to be significant. With pitchers and hitters given different daily weights.
vr, Xei

Wobatus
Guest
Wobatus
1 year 11 months ago

I wonder what the distribution was. Did 33 players who were hot continue to outperform, 33 performed to their projections, and 33 fell flat on their face, so on average, they performed to their projections?

X
Guest
X
1 year 11 months ago

I know I can trust the predictions, because they always come with uncertainties or confidence intervals and the extrapolations can be compared to actual data giving a chi-squared per degree of freedom of 1. Oh, they don’t? Then I don’t trust them, do I?

Grant
Guest
Grant
1 year 11 months ago

Sorry, but what does this article and these projections really tell us? If the projection system misses on 2 players in opposite directions but equal amounts, wouldn’t the mean projection look good, but in reality the projection system would be wildly off?

If I predict that Dustin Pedroia will have a .400 woba and Xander Bogaerts will have a .300 woba the rest of the season, and in reality they flip those results (Bogaerts does .400 ROS, Pedroia .300), wouldn’t my projection system be exactly correct in aggregate, and yet terrible incorrect at the same time?

Nathaniel Dawson
Guest
Nathaniel Dawson
1 year 11 months ago

Not really relevant to this. MGL is taking a group of players that have performed poorly, then examining their future performance. It doesn’t really matter if some wildly over-perform, while others wildly under-perform. It’s the aggregate of the group that’s of importance. The aggregate is showing that players that have significantly under- or over-performed as a group tend to move back toward their previous levels in the future.

Nathaniel Dawson
Guest
Nathaniel Dawson
1 year 11 months ago

Also, projection systems are typically tested using Root Mean Square Error (RMSE), which attempts to measure accuracy by measuring how far off each individual projection is, then tallying the results. So using your example of Pedroia and Bogaerts, the projection system would have an average error of 100 points of wOBA*. But neither this article nor MGL’s is about analyzing or measuring projection systems.

*I don’t think that would be the actual result, I just don’t know enough about RMSE to give you the exact number. But that’s the idea.

Grant
Guest
Grant
1 year 11 months ago

I’m no statistics expert (though I do know some), so forgive me if I’m just not understanding the system properly. That said, that RMSE number seems incredibly important, moreso than the result of the projections. Anyone could pick out a pool of 100 players, tally their performance to date, project them to regress to the mean, and then most likely see those results over a large sample size.

To me, the value in projections would be accurately predicting individuals. It doesn’t really seem that insightful to pick a large group of players and observe their inevitable aggregate regression to the mean.

Nathaniel Dawson
Guest
Nathaniel Dawson
1 year 11 months ago

Depends on your definition of “accurate”, I suppose. Right now, there’s no way in hell projection systems are going to be terrifically accurate for all individuals. Maybe in the future, we will have better information that will allow us to be incredibly accurate when projecting a player’s season. I kind of doubt it though. I”m pretty sure we’ll continue to get better, but we can’t ever know with certainty how any player’s future will go. In large part, it seems, because there’s a great deal of random occurrence that’s inherent in the game of baseball, which would result in what we see in MGL’s study. And which would make it incredibly hard to pick out a future Jose Bautista, Chris Davis, of Cliff Lee.

Basic takeaway is projection systems do a pretty good job for the vast majority of players, but they’re always going to miss some. Until and unless some genius comes along that has some way of identifying those players for us.

Mr Scout
Guest
1 year 11 months ago

Concepts very well articulated here…there are fewer people every year that doubt the accuracy of the projections, even hot/cold players. But what is increasingly of interest, now that we’ve got a handle on the big data, are the individual breakouts, like Lee/Bautista/C Gomez, etc… The projection systems are not good at identifying these guys and many people are unaware of this. Thanks for stating it clearly, projection systems are not good at predicting true breakouts.

Great stuff.

Mr Scout
Guest
1 year 11 months ago

Projection systems would have put almost zero players in the HOF…even during HOF players prime, they are regressed to the hall of very good, year after year.

Wobatus
Guest
Wobatus
1 year 11 months ago

Then why is Mike Trout projected to post 5+ WAR over the next 90 games?

Mr Scout
Guest
1 year 11 months ago

Because I said almost zero

Wobatus
Guest
Wobatus
1 year 11 months ago

Doh! Trout’s like the Hall of One.

Bo Jackson
Guest
Bo Jackson
1 year 11 months ago

Mike Trout is the exception to Baseball rules.

Jason B
Guest
Jason B
1 year 11 months ago

…which makes sense. You shouldn’t often project someone to be a top 1-2% player year after year after year. It happens, but there’s a lot more meat in the middle of that bell curve. (i.e., you’ll miss a lot less often predicting a 1-2% player to slide into, say, the 20th-30th percentile than vice versa)

Only glove, no love
Member
Only glove, no love
1 year 11 months ago

What about temporal length of the deviation?

If one can predict (even roughly) the length of the deviation it would be very useful to managers and resolve the whole “play the hot hand or stick with the better player” issue…

Jason B
Guest
Jason B
1 year 11 months ago

I would imagine the ends of hot and cold streaks are generally only visible in hindsight. “Oh of course he was bound to break out! He was 4 for his last 50, he was OVERDUE!”

PackBob
Guest
PackBob
1 year 11 months ago

It seems that the way the projections are constructed and presented they will be very good at predicting the whole group, less so at predicting individual performance. On an individual basis, it would be interesting to see what the distribution looks like.

It also appears that some players stay close to their mean performance while others fluctuate between hot and cold periods. It may not be worthwhile to tease out, but for some players dwelling almost as an outlier on either side of their mean performance seems to be their normal.

Bip
Member
Member
Bip
1 year 11 months ago

I think people can be turned off due to the idea that, for any individual player, the projections most likely will not look that good. If you have a specific player in mind, there is still no way to say with any certainty what he will do for the rest of the season, or the rest of his career. I don’t think people usually want to know if the collection of players who had a hot 1st month will regress, they want to know if the star player on their team is going to come back to normal, or if the hot journeyman is going to keep it up.

The right thing to say to the person is, of course, that the player is much more likely to perform to his projection going forward than he is to maintain the same unusual pace, but he also is not that likely to conform very precisely to the projection either. It’s not something fans are usually interested in hearing.

Dayton Moore
Guest
Dayton Moore
1 year 11 months ago

That’s what I keep telling y’all!! “Trust The Pro-“…wait, whut?

Xeifrank
Guest
1 year 11 months ago

I guess it really depends on what you are using the projections for. There are certain uses that need to be able to pick up on break outs and break downs or whatever you call a player getting worse. If all you are using these projections for is doing some kind of WAR study on which team is the best or helping you to draft the best fantasy baseball team then the current projection systems are more than fine. But if you are using projection systems to beat Vegas with (betting) then you need a projection system that properly weights the more recent stats vs the old stats. A 10 strikeout performance today tells you more about true talent level than a 10 strikeout performance 3 or 4 months ago let alone 15 to 16 months ago. I think everyone agrees with that. The question is what is the proper weighting. Without giving out my proprietary data I am going to say that projection systems do two things. They regress too much and they under weight the most recent data by quite a bit.
vr, Xei

No
Guest
No
1 year 11 months ago

No, everybody doesn’t agree that a recent performance automatically tells you more about true talent level than an older one. There are a ton of factors that could contribute to any differences that have nothing to do with time.

Team faced, park, and weather conditions are three easy ones off the top of my head. There are literally countless others.

Your premise is bad and you should feel bad.

Yes
Guest
Yes
1 year 11 months ago

Your attitude is bad and you may be a douche.

Jackelder
Member
Jackelder
1 year 11 months ago

Dave, if you played fantasy baseball you might understand how difficult it is to bet for well projected players who have stunk for eleven weeks. I find it much easier to bet against poorly projected high-flyers doing well. But in every league I’m in it hurts when a rival picks up a player I’ve considered adding, decided he lacks the talent partly because he projected poorly, and then watch that team beat you while Joe Bflstk becomes ROY or an All Star.

I think you should play a couple of Yahoo or ESPN leagues incognito so you can know what we, your readers, feel.

Josh B
Guest
Josh B
1 year 11 months ago

He’d sooner kick your dog for suggesting it.

Smart guy
Guest
Smart guy
1 year 11 months ago

I knew that Joe Bflstk was going to break out all along. Saw that one coming a mile away…

obsessivegiantscompulsive
Guest
1 year 11 months ago

What I’m getting out of this article is that when fans complain about their managers favoring veterans who have “proven” performance levels over unproven young players (forgetting that the manager has played some young players over vets, just not the ones the fans were in love with), even though the vet is producing horrible at the moment, the fans are failing to realize that most players revert to their projections, and hence the better bet over time is to go with the vet with proven production, even if they have not been producing that well that season. Unless, of course, the manager decides to start the young guy instead…

MGL
Guest
MGL
1 year 11 months ago

There is nothing new or groundbreaking or even particularly interesting about this work. I am just trying to get writers, fans, analysts, etc. to STOP assuming that what a player has done so far in the season has any relevance whatsoever beyond their projection. That’s all you read and hear – how so-so-so i batting or pitching THUS FAR IN THE SEASON, and how, because of that, they should be given less or more playing time, or moved up or down in the order, or some such thing. Or how so-and-so team is “for real” or “not for real” based on their underlying statistics. That kind of “analysis” is crap. The ONLY thing that matters for those things are projections. I was merely trying to illustrate that.

To tell you the truth, even though I deal with baseball analysis and player projections and valuations almost every day of the year on some level or another, I never look at how a player has done so far during the season. It doesn’t interest me at all.

Fun Guy
Guest
Fun Guy
1 year 11 months ago

That doesn’t seem like a very fun way to enjoy baseball.

AlbionHero
Guest
AlbionHero
1 year 11 months ago

Are you even a baseball fan? Or just obsessed with numbers?

Josh B
Guest
Josh B
1 year 11 months ago

Projections are a great starting point and take into consideration a lot of important information, which I can then apply my own judgment to. The speed limit is a nice starting point too, which takes into consideration the results of a lot of safety testing (I assume), but you then apply your own judgment to that and generally go 1-10 miles above it (don’t you?). I used my brain to tell me that Garrett Richards increased velocity and command were going to serve him well on the high seas this year. The projections, much like the scarecrow from the Wiz, has not a brain, and therefore failed to make the same prediction that I, a brain-having human, did.

Tom
Guest
Tom
1 year 11 months ago

Wow. You da man!

Smart guy
Guest
Smart guy
1 year 11 months ago

Yep, Josh will be winning those $1,000,000 prizes for top fantasy performer in the country aaaaany day now. (Right??)

…or Josh is conveniently picking and trumpeting his success while conveniently staying mum about any poor predictions. (Right?? Right.)

Frank
Guest
Frank
1 year 11 months ago

If I take every player in baseball and assume they’ll all regress slightly toward league average, then consider them only in aggregate, I’m right every year!

MGL
Guest
MGL
1 year 11 months ago

What a dumb comment. Go jump in a lake, Frank.

Frank
Guest
Frank
1 year 11 months ago

Okay… wait a minute! You’re a moron. In real life.

G.Hahd
Guest
1 year 11 months ago

He’s a sociopath what do you expect.

Blootzkloof
Guest
Blootzkloof
1 year 11 months ago

“Go jump in a lake, Frank”

Intelligent comment

Frank
Guest
Frank
1 year 11 months ago

Seems legit.

Jake in Pittsburgh
Guest
Jake in Pittsburgh
1 year 11 months ago

“(F)orecasts…certainly didn’t anticipate Jose Bautista [Edwin Encarnacion](Juan Francisco) {Melky Cabrera} was going to go from career scrub [above average] (sub-replacement) {marginal 3rd/4th OF} to superstar [slugger] (budding slugger) {confirmed PED user & budding slugger} at the age of 29 [29](27) {29}” in Toronto.

Amazing.

Here’s a projection: Just like Bautista and Encarnacion, Francisco and Melky, if they play all year, will have their first 30-HR campaigns this year. In Toronto.

Amazing.

I’ll tiptoe out now, so you all can close your eyes again, snuggle up with your teddies, and go back to talking about “outliers”.

Jason B
Guest
Jason B
1 year 11 months ago

A) Neither Melky nor Francisco are on pace for 30 HR.

B) Melky’s ISO (.167 as of 06/16) is right in line with 2011 (.164) and 2012 (.170)

Amazing!

Craig Biggio
Guest
Craig Biggio
1 year 11 months ago

I make $12,000 a month working from home! Ask me how you can too! (Millions in the bank = $12,000 a month interest.)

Jonah Pemstein
Member
Member
1 year 11 months ago

What about players who go a whole season overperforming/underperforming? Are they in line for regression the next season?

Blootzkloof
Guest
Blootzkloof
1 year 11 months ago

See, I don’t think that projections work well for individual players. It’s true that projections even out over large sample sizes, but a lot of things even out over large sample sizes. For example, I can predict that the average runs per game this year in MLB will be 4.15. I did not use any math or science, I just looked at what it was last year (4.17). Now if that 4.15 mark holds up (it’s 4.14 now), that don’t necessarily make me Nostradamus. That don’t mean I can predict how many runs the Houston Astros will score against the Mariners on July 1, 2014. If I predict that the Astros will score 4.15 runs on July 1, I’m probably going to be mad wrong, you get me?

Seriously, the projection systems are not geniuses just because they can predict large sample sizes. Any fool with a calculator and half a brain can predict a large sample size, that don’t mean we should trust them.

Los Tigres
Guest
1 year 11 months ago

But can you beat their RMSE?

Blootzkloof
Guest
Blootzkloof
1 year 11 months ago

Now if y’all don’t mind, I’ll go back to enjoying Edwin Encarnacion, Jose Bautista, Chris Davis, Charlie Blackmon, Nelson Cruz, and Yangervis Solarte. I’ll do it while laughing over Bryce Harper’s projected .905 OPS and Prince Fielder’s 30 homeruns that was predicted by ZiPS

Jason B
Guest
Jason B
1 year 11 months ago

I’m sure your prediction systems would work a lot better? Do share.

lanceomatic
Guest
lanceomatic
1 year 11 months ago

The inclusion of Yangervis Solarte is what makes that list really special. He’s pretty much already regressed back to his projections, only significantly exceeding them in one area, BB%. He might be this year’s Daniel Nava, flashing power and OBP early only to regress to nothing special.

And Charlie Blackmon’s AVG by month .374, .260, .250. So yeah have fun owning those guys in your fantasy leagues everyone.

Yeah obviously there are guys that have altered something in their game and so now they are better/worse than their projections but those of us who just pay a little attention know those guys are the outliers and they are really somewhat rare.

wpDiscuz