# The Differences Between Predictions and Projections

In my Monday post about the White Sox recent success beating preseason projections, I included a statement that I’ve mentioned a few times over the last few years:

But also, please just keep in mind that projections are not predictions. They are a snapshot of what we think a team’s median true talent level might be, and it should be understood that there’s a pretty sizable margin for error based on things that projection systems simply can’t forecast, and also the errors that come from having imperfect information or imperfect calculations.

I wrote about this distinction a couple of years ago, but I think it’s worth delving into the differences again. For one, FanGraphs has gotten a lot larger over the last few years, so many of you might not have read that piece, but also, I think there’s a few things that I could have stated better in that article, and I want to give more context for why I see the distinction as meaningful rather than being a semantical argument with no practical use.

Let’s start out by acknowledging that predictions are a subset of projections. Or, to put it another way, predictions are projections, but a projection isn’t necessarily a prediction. I know that’s a bit of a tongue twister, and seems like a semantical difference, but think of it like this: Mothers are women, but not all women are mothers. No one would suggest that it is simply semantics to clarify whether a women is or is not a mother. There’s a meaningful difference there.

So it is with predictions and projections. A prediction is essentially a projection where there is a high degree of confidence in a specific outcome. Not all projections lead to that kind of confidence in one result, however. In fact, in many cases, an accurate projection will result in a range of outcomes where there is no single result that is likely to occur.

Let’s take the NBA’s Draft Lottery, for instance. The 14 non-playoff teams get various combinations of numbers assigned to them, and those numbers correspond to 14 ping pong balls that are placed into a lottery machine. The team with the worst record gets 250 of the 1,000 possible combinations, and then the second worst team gets 199, and each successive team gets fewer than the one in front of them, down to the 14th worst team getting just five of the 1,000 possible combinations. Because the NBA doesn’t want teams to drop too far by random chance, they only draw for the first three selections, and then the remaining teams are slotted in from #4 to #14 based on win-loss record in the previous year. The Wikipedia entry on the draft lottery has a pretty nifty chart showing the various odds of each outcome, which we’ll reproduce here:

Seed | Chances | 1st | 2nd | 3rd | 4th | 5th | 6th | 7th | 8th | 9th | 10th | 11th | 12th | 13th | 14th |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

1 | 250 | .250 | .215 | .178 | .357 | ||||||||||

2 | 199 | .199 | .188 | .171 | .319 | .123 | |||||||||

3 | 156 | .156 | .157 | .156 | .226 | .265 | .040 | ||||||||

4 | 119 | .119 | .126 | .133 | .099 | .351 | .160 | .012 | |||||||

5 | 88 | .088 | .097 | .107 | .261 | .360 | .084 | .004 | |||||||

6 | 63 | .063 | .071 | .081 | .439 | .305 | .040 | .001 | |||||||

7 | 43 | .043 | .049 | .058 | .599 | .232 | .018 | .000 | |||||||

8 | 28 | .028 | .033 | .039 | .724 | .168 | .008 | .000 | |||||||

9 | 17 | .017 | .020 | .024 | .813 | .122 | .004 | .000 | |||||||

10 | 11 | .011 | .013 | .016 | .870 | .089 | .002 | .000 | |||||||

11 | 8 | .008 | .009 | .012 | .907 | .063 | .001 | .000 | |||||||

12 | 7 | .007 | .008 | .010 | .935 | .039 | .000 | ||||||||

13 | 6 | .006 | .007 | .009 | .960 | .018 | |||||||||

14 | 5 | .005 | .006 | .007 | .982 |

If you were the only person on the planet who knew those odds, what kind of predictions would you be willing to make? Would you predict that the team with the #1 seed would win the first overall pick? I’d hope not, because you’d be wrong three times out of four, even though you were selecting the most likely outcome every single time. On the other hand, you probably would be willing to predict that the #14 seed would pick 14th, because a 98.2% chance of being right is pretty darn good.

Depending on how much you value your own credibility, you might even be willing to predict the outcome of picks #8 through #13, since the likelihood of being right on each was greater than 72%. You wouldn’t always be right, but you’d be right often enough that your overall record would come out looking pretty good. But, hopefully, you’d be wise enough to steer away from predicting any kind of specific result for anything in the top 7, where your odds of being right would be between 25% and 60%, meaning you’d be taking the side of something close to (or worse than) a coin flip in each case. If someone asks you to predict who is going to win the #1 overall pick in the NBA Draft Lottery, a correct interpretation of the data is simply “I don’t know.”

Preseason win-loss projections for Major League teams are much like the NBA draft lottery, just with the caveats that we’re not dealing with perfectly known variables and there’s no artificial floor placed below each team to keep them from crashing due to random variation. With all of the unknowns that are simply outside of the realm of forecasting, every possible win-loss record you could dream up for any team is unlikely. It doesn’t matter how good or how bad the team is; the spread of talent across the league is simply not large enough to allow us to have confidence in any given win-loss record to make a prediction, given all of the variables that we know we can’t forecast with any kind of certainty.

It doesn’t mean that these forecasts are useless, of course. Despite having a range of unlikely outcomes, we can still come up with a projection that is likely enough to occur for us to make a prediction, but that projection has to be a range of numbers, not a single outcome. Since even the best projection systems tend to have standard deviations from actual win-loss results of 6-10 wins, we can say with something like 95% confidence that a team will finish within +/- 16 games of their mean projection. So, you could confidently predict that a team that has a projected 81-81 record would win between 65 and 97 games.

The problem, of course, is that’s not very helpful. Anyone could predict that any team will be somewhere between “terrible” and “excellent”, and you certainly don’t need any fancy algorithms to say that a team could finish somewhere between first and last. This is why making preseason predictions is kind of silly. We simply don’t know enough in advance to be confident enough in our forecasts to make declarative statements about small ranges of outcomes.

We don’t have to get to the 95% confidence level that two standard deviations brings about, of course. Knowing that 68% of teams fall between +/- 8 wins of their projected record is still useful, as long as the results aren’t overstated. Knowing that, we can look at a team with a projected 75-87 record as an unlikely contender, but more importantly, we can look at a group of six teams projected for mid-70s records and realize that one of them will probably make a playoff run, since we’d expect two of the six teams to fall outside of the standard deviation range, with one on the high side and one on the low side.

In other words, if we look at all the teams that are projected to win between 75-80 games, we might find a list that includes the Orioles, White Sox, Brewers, Pirates, Padres, and Royals. None of them are likely to make the playoffs, but as an aggregate group, this is a pretty good place to start if you’re looking for a “surprise team” in 2013. It doesn’t mean that the surprise team will certainly come from that group — the Orioles weren’t forecast as a mid-70s win team last year, for instance — but starting with the preseason forecasts and knowing the standard deviation can help guide decisions about what teams should be making more aggressive efforts to improve their teams in the short term versus focusing on the bigger picture.

Where one can start to get into trouble is if they start treating all projections as if they’re predictions. Every preseason win-loss forecast that comes out over the next six weeks is going to put a single number on each team as the most likely outcome, but it’s important to remember that every single of those numbers is likely to be wrong, and that the spread in expected wins around that number is pretty large. When a team like the Indians starts upgrading their roster, the hope is not that they can push their forecast mean total up to 81 wins from 75 wins — which can be viewed as a meaningless difference if one is solely focused on a binary playoffs/no playoffs outcome — but that they can raise the amount of opportunities they have to have things break right and end up with 90+, sneaking their way into October baseball in the process.

The conflation of projections and predictions lies partly with the public’s fascination with “making a pick” and then defending it — those kinds of stories are extremely popular and drive a lot of traffic — but are also born out of the way forecasters have chosen to display their results. If we want to really get across the meaningful difference between projections and predictions, maybe we’d be better off displaying the results of preseason projections as overlaying bell curves rather than a simple standings table with the weighted mean representing the entire projection. Or maybe something like the way the guys at RLYW do it, with pie charts showing the differences in how often a division is won by each team in its simulations.

So, forecasters, here’s my request: Show us more than the single weighted mean outcome when doing win-loss records. Give us the confidence level of each number between 60 and 100 wins. That’s interesting data, and it’s helpful in pointing out that the projections you’re making are not predictions that you’re attempting to stake your reputation on. And, writers quoting those projections, let’s do a better job of calling them what they are. Or, more specifically, what they aren’t. The forecasters are doing a real service by publishing their results. Let’s not pretend that all that work is simply a prediction, no different than a random number pulled out of thin air by a television talking head. There is a difference, and we should try to shine a spotlight on those differences whenever possible.

Print This Post

Good article. And I think this is a great explanation of your often stated point that teams should continue to add talent even if they are low or in the middle of the win curve. Frequently people are thinking that if you aren’t projected to win 90+ you should blow it all up and start over, which is ridiculous. Very relevant to your arguments for the Mets to sign both Dickey and Wright to extensions as well.

Speaking of the RLYW guys:

Dave…if you have a connection there, please inform them that they have double counted Freddy Garcia in the most recent CAIRO projections. (He’s listed twice, which may be screwing up their projected standings.) I can’t find contact information.

Thanks.

Garcia accidentally got listed twice in the projection spreadsheet, but that is not an issue with the projected standings which are done using a simulator.

They’re double counting a ton of pitchers from what I’ve seen. Probably over 100. Most of the double counts have different RAR/WAR values but other than that the entries are identical. I was curious about this, myself.

Looks like a problem with how I generate the depth charts. Some guys are getting pulled in multiple times. I’ll fix it in the next release but the projected standings aren’t impacted by it.

I fully agree that we need to do better about showing the uncertainty inherent in projections. However, I personally don’t differentiate between “prediction” and “projection” in the same way you do. I would use the words interchangeably, but the good ones will have the appropriate levels of uncertainty/confidence. For instance, I would in fact predict that the #1 seed would win the #1 pick, and my prediction would have 25% confidence. I’d also mention the chances for the other teams. In my mind, a prediction/projection *is* a probability distribution, and it’s harmful not to treat it as such.

If you want to make that distinction in language that’s fine too though, but the important part is for us to think probabilistically.

Well, the issue is that the words mean different things within different communities of practice. Sometimes a projection is a subset of prediction, sometimes a prediction is a subset of projection! It literally depends on the group of people you’re talking with.

So, to amend Dave’s mother/woman concept, imagine that the world become cohabited by aliens where all women were mothers but not all mothers were women. Now try to disentangle the terms. Basically, Dave is stating that within this community of discourse, he’s treating predictions as a subset of projections. Which is fine, since it beats having the terms be so ambiguously related that they’re both almost useless.

And, for the record, I mentally define prediction and projection somewhat differently. Coming from a systems and modeling standpoint, I consider a projection something that extrapolates from the known data (e.g., given prior events, project paths of future events). A projection can’t be “wrong”, as it’s just the mechanistic result of applying a model. In other words, projections are data-driven.

Prediction, on the other hand, is meant to be tested. A prediction can be right or wrong. In other words, predictions are results-driven. I’d say that’s qualitatively different from being data-driven. For example, let’s say you have a savant who can actually predict the NBA draft lottery better than chance, just by looking at the ball machine. If you’re doing predictions, you’d be a fool not to consult him. How would you build that into a projection? “Well, we calculated the exact odds based on the probabilities, and then added a vector for Rainman’s predictions that improves the correlation by 10%…”

Heck, some of the best predictions are self-fulfilling prophecies. By comparison, calculating a projection and then doing everything in your power to increase the fit? Generally frowned upon. Probably for good reason (“I’m sorry Aubrey Huff, but my simulation projected that you would spend 90 days on the DL this year. Hold very still…”).

So I generally lean toward a view that a projection is a systematic approach for modeling data, which produces some estimates related to future events. On the other hand, I’d say that predictions are characterized by their results (right/wrong, close/no cigar). Whether these results are arrived at by advanced statistics or astrology, they’re still predictions. However, if your advanced stats can’t outperform astrology, it may be time to find a new line of work (palm reading maybe?).

Give us the confidence level of each numberThis is great advice for projections of individual players as well.

I think BP used to provide the 10th, 25th, 50th, 75th and 90th percentiles for their player projections …

I hope people pay attention to the last paragraph

Mickeyg13 hits the nail on the head. The main point is that we need to quantify our uncertainty. Good predictions (or projections) should have probability distributions around them, reflecting the confidence of our claim.

I also concur with Mickeyg13 and Beef. Perhaps it is because I work in statistics, but I don’t think the word *prediction* implies any greater certainty than does the word *projection*. A projection is a prediction. I suppose it is called a projection because it is a prediction of a future value, but it is still a prediction. All predictions have uncertainty and that uncertainty should be presented along with the predicted value.

If 68% confidence interval is +-8 wins, 95% is +-24 wins.. Just sayin’. Even more illustrative of your point. Great post though

I think you made a mistake there. To go from 68% to 95% you go from one deviation in either direction to two, so it doubles the spread, not triples it.

I’m not a woman, I am a nation.

95% is 3 standard deviations. 1=68%, 2=90%, 3=95%, 4=99%

Nope: http://en.wikipedia.org/wiki/68-95-99.7_rule

My mistake… Nice catch

I suspect that the Dodgers will be a excellent example of this point, lots of talent may not equal success.

I think you may be right. I know we talk about tangible stuff around here but the intangibles on that team scare the poop out of me.

I have my own projection system (7.6 std dev over 3 years) and do exactly what you recommend- I use my past error as a probability distribution to find each team’s odds of winning their division.

Frankly, the only thing that’s missing (at least not readily apparent) from most sites/people’s projection estimates is the uncertainty. It would be more informative if everyone just stated a win projection with ± estimate (could be StDev, or St. error; ideally it’d be 95% Confidence).

Unless you spend a fair amount of time using statistics people seem to prefer a set number. It’s also less visually attractive.

Great explanation! This might be Dave’s best post in quite a while.

I hope your projection is way off.

It seems odd to criticize others for not publishing standard deviations when this website also does not publish standard deviations.

FanGraphs doesn’t do their own projections, they simply display the projections of outside sources, so everything he said in his piece would apply equally to the projection systems they display here.

Because SD is dumb?

I appreciate the general thrust of this article, but I don’t think it goes far enough. What’s the utility of distinguishing between “projections” and “predictions”? In other words, why say something has “high” certainty when we can actually quantify the uncertainty? Whether we think certainty is “high” or “low” should emerge from the interpretation of the data. All estimates are inherently uncertain. It’s simply a question of how much exists. “High” is a very imprecise and arbitrary description of how much uncertainty exists for an estimate, and it’s unnecessary to be so imprecise.

“All estimates are inherently uncertain.”

Funny, that’s what my mechanic said too.

I accidentally downvoted your comment, but I want you to know that it gets a +1 from me.

Now two kind people who didn’t like the comment need to give it a + anyway to make up for this. Please do.

It’s a shame prospect evaluation is so far behind the rest of baseball analysis when it comes to this.

I heartily agree with your central point: That there are so many uncertainties in predicting win totals that forecasters would do better to show a range and provide confidence levels at different intervals.

But I also believe that you obscure that point by your use of language, specifically, the supposed distinction between a projection and a prediction. Those two terms are used in a variety of fields and carry different meanings within those different fields. If you limit the definition to the field of statistics, your definition fails to capture precisely the difference; among many statisticians, a prediction is based solely on a past data set that allows one to calculate a future data point without the need to make any additional assumptions; a projection requires additional assumptions. For purposes of sabermetrics, we are almost always dealing with additional assumptions and therefore projections.

“This is why making preseason predictions is kind of silly.”

Even sillier is when someone makes a prediction that happens to happen, and then they crow about their baseball acumen.

I’m glad someone finally mentioned how bad fangraphs war is at predicting wins. The thing is though, that +/- 16 is not how bad at is at predicting wins in the preseason, but how bad it is when you know the records of the teams and have a full season of data. This raises the question of why baseball-reference doesn’t seem to have this problem.

Of course, that’s not what fWAR is meant to be used for, so (as was said on another discussion thread) if you use the wrong tool for a particular job, don’t blame the tool. Don’t blame that flathead screwdriver because it’s not hammering very well.

Always love when someone gets a prediction “right” and becomes viewed as an expert

Great article Dave.

This is exactly the reason why I said “We are going to compete for a championship.” Since the standard deviation is around 8 wins and we are projected to have around 81 wins, we could end up with 89 wins which potentially maybe could win the division. There is about 17% of that. Also, we have to factor in the chances of the other NL Central teams not reaching 89 wins. I’ll estimate that at around 15%. Then, I’ll just estimate the potential of winning the World Series after making the playoffs at around 12.5%. So the Pirates do have a chance at a World Series this year. It’s the almost not insanely small .32%… Good thing I know my rhetoric to excite Pittsburgh fans.

Technically, don’t all teams compete for a championship?

In 2013, the Astros, Marlins, and Cubs seem to have no interest in doing so.

Words are merely projections of ideas and thus have an inherent margin of error… All joking aside a prediction is most certainly not “a projection where there is a high degree of confidence in a specific outcome” people make predictions all the time with little to no confidence. In your own words later in the article you refer to a prediction as a “random number pulled out of thin air by a television talking head”- so a prediction can’t be both right?

I think the main idea of the article that there exists a type forecast that was arrived at with math and has a margin of error and that there also exist forecasts that might be more specific in nature and they probably have less to do with math and more to do with something like “gut”. Call these things whatever you want but don’t confuse them as the same thing.

When do the Pecota standings thingymajigs come out?

Version 1.0102a is out now!

Can’t wait for the idiots on the sports talk shows to take up this advice.

“I’m not saying who’s going to win the NL East, I am telling you, right now, that the Nationals have a 43% of doing it!”

“forty-three! Forty-three?!? You are sadly mistaken by at seven percent my friend. I would say sixty-three if Dan Haren is healthy.”

“I didn’t say anything about Dan Haren yet, but if he’s healthy and Bryce Harper can do what Mike Trout did, nothing can stop them ninty four and a half percent of the time.”

I don’t see a philosophical difference between the 75% probability of being wrong on the top pick and the 1.8% probability on pick 14. They are both just predictions with different degrees of uncertainty.

If Dave wants to nipitck on that, then he should tell us what would be the cutoff between a projection and a prediction? 50%? 75%? 98%?

And if the only way of calling it a prediction would be to be close to 100% sure of it, you could just call it a fact.

You’re still misunderstanding slightly. We need to stop thinking in terms of “right” and “wrong” or even “wrong 75% of the time” but in terms of a probability distribution: “The value of x will fall in range R p% of the time.” Dave wants to say that predictions are the subset of projections where R is small and p is large.

So predictions/projections are like the positions of electrons–a probability cloud. I can dig that.

I agree with the sentiment, I just find the use of the distinction between projection and prediction absurd.

If we use that type of definition, than we should be clear on what are the cutoff R and p% for one or the other. Otherwise it is just a semantic discussion that adds nothing.

Let me take a shot at applying Dave’s set-subset example.

Carson Cistulli is a woman

Carson Cistulli is not a mother

I project that Carson’s next prospect review will feature a under-talented and under-hyped prospect*

I predict that Carson’s next prospect review will feature a under-talented and under-hyped prospect

*the standard deviation in this prediction is 0, it will always be true.

I am not sure if I am heading in the right direction here.

This is so much worse than I interpreted his original article. You’re basing the difference on an arbitrary distinction of “high confidence” (whatever that means).

I’d have to support Taleb to find such pretentious absurdity.

Atleast the end of the article points out some aspects that aren’t semantic arbitrary distinctions. It should be pointed out that using SD is pretty laughable as is the idea of a bell curve. A team with a median of 93 wins IS MUCH MUCH MUCH more likely to hit 73 wins than 113 wins, but why actually write useful stuff. Though there are teams with lower medians that certainly have right hand kurtosis. (Wow. We can discuss other central moments!!!)

Poseur alert!

This is exactly what I was going to say, but you beat me to it.

Dave, this sounds a bit like a very mini version of parts of Nate Silver’s recent book “the Signal and the Noise”. I know you’re familiar with some of his work, and have posted on him previously, you’ve read the book, I suspect? The book’s brilliant in explaining some of this stuff, and other aspects of statistical analysis.

Like the article, but I’m in the camp that the projection/prediction distinction is too contrary to common usage for my liking. It seems more instructive to say that predictions have some uncertainty attached with them, and the good predictors are the ones who can most accurately quantify this uncertainty.

This notion really got lost in the whole Nate Silver election predictions. People judged his results on how many states he “called”, but really we should be judging on how closely the real percentages matched his given percentages. Dave hit the nail on the head when he says it’s all about “making a pick”. If you try to bring uncertainty into the discussion it’s considered a cop out. It’s an annoying phenomenon.

I’d love to see probability charts on what % chance team A will win 81+ games, etc. For that matter, it would be great to see the % chance that player X will exceed a WAR of 2, or 30 HR, etc., rather than just a prediction of the expected mean.

The intent of this article is very important and articles on it ought to be written on this site from time to time. I especially liked the NBA lottery example. People should realize that all statements or statistics about the future are probablistic at best, but they don’t.

The prediction/projection definition was unfortunate and the suggestion that all projections be shown as probability charts is, of course, totally unworkable.

Nevertheless, keep sounding the trumpet.

Great article, enjoyed reading it very much.

wrr