Good article. And I think this is a great explanation of your often stated point that teams should continue to add talent even if they are low or in the middle of the win curve. Frequently people are thinking that if you aren’t projected to win 90+ you should blow it all up and start over, which is ridiculous. Very relevant to your arguments for the Mets to sign both Dickey and Wright to extensions as well.
Comment by Caveman Jones — February 20, 2013 @ 4:41 pm
Speaking of the RLYW guys:
Dave…if you have a connection there, please inform them that they have double counted Freddy Garcia in the most recent CAIRO projections. (He’s listed twice, which may be screwing up their projected standings.) I can’t find contact information.
I fully agree that we need to do better about showing the uncertainty inherent in projections. However, I personally don’t differentiate between “prediction” and “projection” in the same way you do. I would use the words interchangeably, but the good ones will have the appropriate levels of uncertainty/confidence. For instance, I would in fact predict that the #1 seed would win the #1 pick, and my prediction would have 25% confidence. I’d also mention the chances for the other teams. In my mind, a prediction/projection *is* a probability distribution, and it’s harmful not to treat it as such.
If you want to make that distinction in language that’s fine too though, but the important part is for us to think probabilistically.
Mickeyg13 hits the nail on the head. The main point is that we need to quantify our uncertainty. Good predictions (or projections) should have probability distributions around them, reflecting the confidence of our claim.
If 68% confidence interval is +-8 wins, 95% is +-24 wins.. Just sayin’. Even more illustrative of your point. Great post though
Comment by EJ Johann — February 20, 2013 @ 6:11 pm
I’m not a woman, I am a nation.
Comment by Mother Russia — February 20, 2013 @ 6:19 pm
I also concur with Mickeyg13 and Beef. Perhaps it is because I work in statistics, but I don’t think the word *prediction* implies any greater certainty than does the word *projection*. A projection is a prediction. I suppose it is called a projection because it is a prediction of a future value, but it is still a prediction. All predictions have uncertainty and that uncertainty should be presented along with the predicted value.
Comment by Keith Karcher — February 20, 2013 @ 6:19 pm
I think you made a mistake there. To go from 68% to 95% you go from one deviation in either direction to two, so it doubles the spread, not triples it.
I think you may be right. I know we talk about tangible stuff around here but the intangibles on that team scare the poop out of me.
Comment by Spit Ball — February 20, 2013 @ 6:58 pm
My mistake… Nice catch
Comment by EJ Johann — February 20, 2013 @ 7:15 pm
Frankly, the only thing that’s missing (at least not readily apparent) from most sites/people’s projection estimates is the uncertainty. It would be more informative if everyone just stated a win projection with ± estimate (could be StDev, or St. error; ideally it’d be 95% Confidence).
Unless you spend a fair amount of time using statistics people seem to prefer a set number. It’s also less visually attractive.
It seems odd to criticize others for not publishing standard deviations when this website also does not publish standard deviations.
Comment by Crap Shoot — February 20, 2013 @ 8:13 pm
I appreciate the general thrust of this article, but I don’t think it goes far enough. What’s the utility of distinguishing between “projections” and “predictions”? In other words, why say something has “high” certainty when we can actually quantify the uncertainty? Whether we think certainty is “high” or “low” should emerge from the interpretation of the data. All estimates are inherently uncertain. It’s simply a question of how much exists. “High” is a very imprecise and arbitrary description of how much uncertainty exists for an estimate, and it’s unnecessary to be so imprecise.
Well, the issue is that the words mean different things within different communities of practice. Sometimes a projection is a subset of prediction, sometimes a prediction is a subset of projection! It literally depends on the group of people you’re talking with.
So, to amend Dave’s mother/woman concept, imagine that the world become cohabited by aliens where all women were mothers but not all mothers were women. Now try to disentangle the terms. Basically, Dave is stating that within this community of discourse, he’s treating predictions as a subset of projections. Which is fine, since it beats having the terms be so ambiguously related that they’re both almost useless.
And, for the record, I mentally define prediction and projection somewhat differently. Coming from a systems and modeling standpoint, I consider a projection something that extrapolates from the known data (e.g., given prior events, project paths of future events). A projection can’t be “wrong”, as it’s just the mechanistic result of applying a model. In other words, projections are data-driven.
Prediction, on the other hand, is meant to be tested. A prediction can be right or wrong. In other words, predictions are results-driven. I’d say that’s qualitatively different from being data-driven. For example, let’s say you have a savant who can actually predict the NBA draft lottery better than chance, just by looking at the ball machine. If you’re doing predictions, you’d be a fool not to consult him. How would you build that into a projection? “Well, we calculated the exact odds based on the probabilities, and then added a vector for Rainman’s predictions that improves the correlation by 10%…”
Heck, some of the best predictions are self-fulfilling prophecies. By comparison, calculating a projection and then doing everything in your power to increase the fit? Generally frowned upon. Probably for good reason (“I’m sorry Aubrey Huff, but my simulation projected that you would spend 90 days on the DL this year. Hold very still…”).
So I generally lean toward a view that a projection is a systematic approach for modeling data, which produces some estimates related to future events. On the other hand, I’d say that predictions are characterized by their results (right/wrong, close/no cigar). Whether these results are arrived at by advanced statistics or astrology, they’re still predictions. However, if your advanced stats can’t outperform astrology, it may be time to find a new line of work (palm reading maybe?).
I heartily agree with your central point: That there are so many uncertainties in predicting win totals that forecasters would do better to show a range and provide confidence levels at different intervals.
But I also believe that you obscure that point by your use of language, specifically, the supposed distinction between a projection and a prediction. Those two terms are used in a variety of fields and carry different meanings within those different fields. If you limit the definition to the field of statistics, your definition fails to capture precisely the difference; among many statisticians, a prediction is based solely on a past data set that allows one to calculate a future data point without the need to make any additional assumptions; a projection requires additional assumptions. For purposes of sabermetrics, we are almost always dealing with additional assumptions and therefore projections.
Comment by Jonathan Sher — February 20, 2013 @ 10:03 pm
I accidentally downvoted your comment, but I want you to know that it gets a +1 from me.
I’m glad someone finally mentioned how bad fangraphs war is at predicting wins. The thing is though, that +/- 16 is not how bad at is at predicting wins in the preseason, but how bad it is when you know the records of the teams and have a full season of data. This raises the question of why baseball-reference doesn’t seem to have this problem.
Comment by Stathead — February 21, 2013 @ 12:33 am
Always love when someone gets a prediction “right” and becomes viewed as an expert
Comment by sprained left fat — February 21, 2013 @ 12:51 am
This is exactly the reason why I said “We are going to compete for a championship.” Since the standard deviation is around 8 wins and we are projected to have around 81 wins, we could end up with 89 wins which potentially maybe could win the division. There is about 17% of that. Also, we have to factor in the chances of the other NL Central teams not reaching 89 wins. I’ll estimate that at around 15%. Then, I’ll just estimate the potential of winning the World Series after making the playoffs at around 12.5%. So the Pirates do have a chance at a World Series this year. It’s the almost not insanely small .32%… Good thing I know my rhetoric to excite Pittsburgh fans.
Comment by Bob Nutting — February 21, 2013 @ 12:57 am
Words are merely projections of ideas and thus have an inherent margin of error… All joking aside a prediction is most certainly not “a projection where there is a high degree of confidence in a specific outcome” people make predictions all the time with little to no confidence. In your own words later in the article you refer to a prediction as a “random number pulled out of thin air by a television talking head”- so a prediction can’t be both right?
I think the main idea of the article that there exists a type forecast that was arrived at with math and has a margin of error and that there also exist forecasts that might be more specific in nature and they probably have less to do with math and more to do with something like “gut”. Call these things whatever you want but don’t confuse them as the same thing.
When do the Pecota standings thingymajigs come out?
Comment by DodgersKingsoftheGalaxy — February 21, 2013 @ 2:12 am
They’re double counting a ton of pitchers from what I’ve seen. Probably over 100. Most of the double counts have different RAR/WAR values but other than that the entries are identical. I was curious about this, myself.
Can’t wait for the idiots on the sports talk shows to take up this advice.
“I’m not saying who’s going to win the NL East, I am telling you, right now, that the Nationals have a 43% of doing it!”
“forty-three! Forty-three?!? You are sadly mistaken by at seven percent my friend. I would say sixty-three if Dan Haren is healthy.”
“I didn’t say anything about Dan Haren yet, but if he’s healthy and Bryce Harper can do what Mike Trout did, nothing can stop them ninty four and a half percent of the time.”
Comment by philosofool — February 21, 2013 @ 8:43 am
I don’t see a philosophical difference between the 75% probability of being wrong on the top pick and the 1.8% probability on pick 14. They are both just predictions with different degrees of uncertainty.
If Dave wants to nipitck on that, then he should tell us what would be the cutoff between a projection and a prediction? 50%? 75%? 98%?
And if the only way of calling it a prediction would be to be close to 100% sure of it, you could just call it a fact.
Let me take a shot at applying Dave’s set-subset example.
Carson Cistulli is a woman
Carson Cistulli is not a mother
I project that Carson’s next prospect review will feature a under-talented and under-hyped prospect*
I predict that Carson’s next prospect review will feature a under-talented and under-hyped prospect
*the standard deviation in this prediction is 0, it will always be true.
I am not sure if I am heading in the right direction here.
Comment by Jaybo Shaw — February 21, 2013 @ 10:29 am
Of course, that’s not what fWAR is meant to be used for, so (as was said on another discussion thread) if you use the wrong tool for a particular job, don’t blame the tool. Don’t blame that flathead screwdriver because it’s not hammering very well.
This is so much worse than I interpreted his original article. You’re basing the difference on an arbitrary distinction of “high confidence” (whatever that means).
I’d have to support Taleb to find such pretentious absurdity.
Atleast the end of the article points out some aspects that aren’t semantic arbitrary distinctions. It should be pointed out that using SD is pretty laughable as is the idea of a bell curve. A team with a median of 93 wins IS MUCH MUCH MUCH more likely to hit 73 wins than 113 wins, but why actually write useful stuff. Though there are teams with lower medians that certainly have right hand kurtosis. (Wow. We can discuss other central moments!!!)
Dave, this sounds a bit like a very mini version of parts of Nate Silver’s recent book “the Signal and the Noise”. I know you’re familiar with some of his work, and have posted on him previously, you’ve read the book, I suspect? The book’s brilliant in explaining some of this stuff, and other aspects of statistical analysis.
Like the article, but I’m in the camp that the projection/prediction distinction is too contrary to common usage for my liking. It seems more instructive to say that predictions have some uncertainty attached with them, and the good predictors are the ones who can most accurately quantify this uncertainty.
This notion really got lost in the whole Nate Silver election predictions. People judged his results on how many states he “called”, but really we should be judging on how closely the real percentages matched his given percentages. Dave hit the nail on the head when he says it’s all about “making a pick”. If you try to bring uncertainty into the discussion it’s considered a cop out. It’s an annoying phenomenon.
You’re still misunderstanding slightly. We need to stop thinking in terms of “right” and “wrong” or even “wrong 75% of the time” but in terms of a probability distribution: “The value of x will fall in range R p% of the time.” Dave wants to say that predictions are the subset of projections where R is small and p is large.
Comment by philosofool — February 21, 2013 @ 12:09 pm
I’d love to see probability charts on what % chance team A will win 81+ games, etc. For that matter, it would be great to see the % chance that player X will exceed a WAR of 2, or 30 HR, etc., rather than just a prediction of the expected mean.
Comment by Delmon Youngs sprained left fat — February 21, 2013 @ 12:09 pm
The intent of this article is very important and articles on it ought to be written on this site from time to time. I especially liked the NBA lottery example. People should realize that all statements or statistics about the future are probablistic at best, but they don’t.
The prediction/projection definition was unfortunate and the suggestion that all projections be shown as probability charts is, of course, totally unworkable.
Nevertheless, keep sounding the trumpet.