Of Projections and Predictions

It’s February, finally, so spring training is just around the corner. Teams are putting the finishing touches on their rosters, filling out benches and bullpens, giving jobs to role players and evaluating the walking wounded. And now that the off-season is mostly finished, you should expect to be inundated with the buzzword of the month – projections.

CHONE. ZiPS. MARCEL. PECOTA. CAIRO. And yes, FANS. (Side note: are we capitalizing it, even though it’s not an acronym? I think we should. It’s not like anyone has any idea what CHONE stands for.) There will be no shortage of projection systems tossing out expected performances over the next month or two.

It won’t stop with just individual players, either. The guys over at Replacement Level Yankee Weblog have already run the CAIRO projections through the Diamond Mind simulator 100 times and posted the aggregate results as projected standings. Spoiler alert – the Yankees are good, the Blue Jays are not.

However, we need to make a distinction: projections are not predictions. Projections are information about what we think we currently know, while predictions are speculation about things that we probably cannot know.

This may sound like semantics, but there is an important difference here, and it’s often lost in the way projections are discussed. Too often, projections are treated as predictions of the future. You’ll see people say things like “CAIRO thinks the Blue Jays are going to only win 67 games this year,” for instance.

But that’s not really true. CAIRO thinks that the Blue Jays are on course to win 67 games, and if they don’t do something about it between now and the end of the season, that is their likely destination. But, like a map, the entire point of a projection is to inform the the user so that he can then alter the course if he so desires.

You would never look at a map and say “I’m traveling east on I-40. This map expects me to drive into the Atlantic Ocean.” The map just informs you that the course you are on will eventually lead to the ocean, and if you decide not to exit, wetness awaits. (Obviously, there’s some hyperbole here, as I realize that the freeway doesn’t end with a pier). The map has no expectation of what will happen. It’s just informing you of the course you are on.

Just like you control where you car goes, so do front offices control where their team goes, to a point. It is quite possible that Alex Anthopolous will look at his team’s internal projections and say, “Hey, we kinda suck; someone go find me a third baseman.” And then, after his assistants find him a third baseman, the team will be better. And people will say that the projection was wrong.

But it wasn’t wrong, because it wasn’t predicting anything. It was giving an evaluation of what was true at the time, to the best of its abilities. Its abilities may be flawed (and how you evaluate projection systems is another post for another day), but the intent was never to suggest what will happen, but rather, what could happen if nothing changes, knowing full well that things will change.

So, if you see a projection that you don’t particularly like, don’t get too bent out of shape about it. It’s just information about a path that a team may currently be on in February. By July, there’s a really good chance that the team will be on a different path, and a new projection accounting for that change in course will be available. Most of all, don’t assume that the people behind the projections hate your team. In fact, if your team gets a terrible projection, you may want to thank the system’s creator – that may just be the information that prods the front office to go out and improve the team, thus proving that the stupid system was wrong all along.




Print This Post



Dave is a co-founder of USSMariner.com and contributes to the Wall Street Journal.


107 Responses to “Of Projections and Predictions”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. Dan Lewis says:

    *F*an *A*rrived *N*umbering *S*ystem

    Vote -1 Vote +1

  2. Carl Robinette says:

    Its like the old Proverb:

    “If we dont change direction, we’ll end up where were going.”
    of course this doesnt make sense, since we always end up where were going.

    Also, i like the idea of referring to FANS as a system.

    “FANS projects CarGo for 3.9 WAR!”

    Vote -1 Vote +1

  3. Isn’t “CHONE” just supposed to be a version of “Sean” ? Otherwise it ‘s probably named after a certain third baseman.

    Vote -1 Vote +1

  4. LittlePanda says:

    Forecast of American/National Statistics

    Fan-Aggregated Neato Statistics

    Vote -1 Vote +1

  5. kickbukt says:

    Are there any resources online that aggregates and averages all of these projection systems? I assume doing so would result in a more accurate projection and minimize any weaknesses in any particular system.

    Vote -1 Vote +1

  6. CircleChange11 says:

    I think the article is splitting hairs at such a severe degree that “it doesn’t matter”.

    Whether it’s by definition projections or predictions is almost an afterthought, because the vast majority treat them as predictions … or at least indicators of upcoming performance.

    Heck, I think even the ‘projectors’ themselves speak of them as basically “individual player predictions”.

    I guess the difference between predictions and projections would be the amount of “gut feeling” and “hunch” sprinkled in.

    But, in reality, given the way they are read & used, they are essentially “predictions”. Common usage often trumps definition (unfortunately).

    Vote -1 Vote +1

    • TCQ says:

      This isn’t a linguistics class; intent is not mutable based on misconceptions.

      Vote -1 Vote +1

      • CircleChange11 says:

        Okay, let me explain my thinking …

        If we were to ask a group of guys to “make statistical predictions for every major league player as to how they think they will perform in 2010″, what would the final result of those ‘predictions’ look like? What would the methodology be for computing/calculating/arriving those predictions?

        My guess is that how one would go about making “mass predictions” would very similar as how they make “mass projections”.

        The projections are used by fantasy players as a statistical indicator (prediction even) of how a player will perform in 2010 (based on how they’ve, and others, have performed in the past), and when we use the projections in discussions, we are using them as defacto predictions.

        I’m not really getting hung up on whether we call them projections or predictions or whether they are intended to be used as projections, only that often times they are used as predictions …. almost so frequently and consistently that they have ‘become’ predictions … or at least our most available and reasonable examples of such.

        We can call them whatever we want and describe their intent, but their usage in decision making (fantasy sports) and baseball discussion is of the very nature of “predictions” (at least for the individual players).

        The projection systems seem to be evaluated by how accurate they are in regards to “projected 2010 v actual 2010″, and that would be how we treat predictions.

        I don’t even view it as “intent v misconception”, but rather in terms of application.

        I still keep coming back to the idea of “if they’re not, essentially, ‘predictions’, then why do we care about them?

        Vote -1 Vote +1

      • TCQ says:

        I don’t really feel I need to argue the main point of the article; Dave already did that, and if he wants to extend it, I’ll leave it up to him, but my issue with your position is this: how projections are used and how they are perceived is an entirely separate issue from how they *should* be used and perceived. One does not inform the other.

        Vote -1 Vote +1

      • TCQ says:

        As an addendum, “projected 2010 versus actual 2010″ is not necessarily how we should be judging projections(I believe this is exactly was Dave was getting at). We judge – or should judge – projections based off of how well they use the information available at the time they were created. Some of this can be gleaned from actual performance, but not all, and that’s the key difference in terminology.

        Vote -1 Vote +1

      • CircleChange11 says:

        TCQ, I agree with everything you and Dave said, even if it appears I do not.

        I guess my perspective is that saying that projections should not be used as predictions, and/or projections should not be measured against the actual performance as being “swimming upstream”, since that’s how people want and do use the information.

        I’m not casting projections into the “Nobel and gunpowder” category in terms of intent versus actual usage, only that projections are so commonly used as predictions, that at this point, it doesn’t really matter what the intent or label on them is, because do and will likely continue to use them as predictions.

        Whether any of us right or wrong will likely have no effect on current or future usage. We all use them as some sort of indication of how a player most likely will perform in the upcoming year. If they did not serve that purpose (intentionally or unintentionally) we would likely not give them much attention.

        Vote -1 Vote +1

      • TCQ says:

        Fair enough.

        Vote -1 Vote +1

      • CircleChange11 says:

        Err, ‘Nobel and Dynamite’, not ‘Nobel and Gunpowder’.

        Vote -1 Vote +1

  7. Llewdor says:

    Even the Replacement-Level Yankee Blog pointed out that the projection doesn’t take into account the ability, for example, of Toronto to produce league-average pitching out of thin air.

    Vote -1 Vote +1

    • SG says:

      Right. It’s really tough to statistically account for things like that, but that doesn’t mean they don’t exist. Same thing with the Angels, who seem to always exceed their pre-season projections.

      Vote -1 Vote +1

  8. Rick says:

    It would be quite helpful if projection systems included their margin or error. Knowing whether its 67 +- 3 or 67 +-15 makes a big difference. Just showing a simple set of histograms, one for each team in a division arranged vertically, would be very helpful in communicating the difference.

    Vote -1 Vote +1

    • SG says:

      It would be quite helpful if projection systems included their margin or error. Knowing whether its 67 +- 3 or 67 +-15 makes a big difference. Just showing a simple set of histograms, one for each team in a division arranged vertically, would be very helpful in communicating the difference.

      Since these are early projections that weren’t meant to be taken seriously, I didn’t give the detail I will give when I run my bigger set of projections with multiple projection systems in March. But that will all be part of it. The 2009 version can be found here:

      (AL) http://rlyw.blogspot.com/2009/04/2009-diamond-mind-projection-blowout.html

      (NL) http://rlyw.blogspot.com/2009/04/2009-diamond-mind-projection-blowout_01.html

      As far as margin of error, with Diamond Mind, the standard deviation is around 7 wins. In actuality it’s probably closer to 10 wins.

      So figure the average win total as your middle bound and you’re looking at a range of Avg W – 10 to Avg W +10. And yeah, that’s not really precise, but anyone who’s looking at these for precision doesn’t really understand the purpose of them.

      Vote -1 Vote +1

  9. alskor says:

    CAIRO… why are we including a “projection system” on a Yankees website along with the other legit systems?

    CAIRO just takes the Marcels and changes the underlying assumptions and components in a bunch of ways that make the Yankees look better.

    They admit themselves that “[t}his is an almost entirely objective system.”

    Maybe I’ll take CHONE and assume all my team’s favorites players will get more playing time and the older players won’t be affected as much by age. Then I’ll “crunch the numbers” again and voila – my team is awesome! 102 wins and 10 more wins than any team!

    I think I’ll call my “projection system” HOMER.

    Vote -1 Vote +1

    • mattymatty says:

      Someone should do something like that for the Royals. They could call the system QUISENBERRY.

      Vote -1 Vote +1

      • CircleChange11 says:

        They could call the system QUISENBERRY.

        THIS just SCREAMS “Acronym Contest”!

        Knowing Quiz’s character, love of language & poetry, and overall quirkiness … he’d love an acronym for “QUISENBERRY”. *silly grin*

        Vote -1 Vote +1

    • Sam says:

      I am hoping this is kind of a joke, but if not, here is a reason why you may want to take another look at CAIRO:

      http://vegaswatch.net/2009/11/evaluating-april-mlb-predictions-2005.html

      In 2009, it was the best projection system. And SG is a first-rate analyst.

      Vote -1 Vote +1

      • alskor says:

        Oh please. That’s the first and only year of CAIRO so far. The Yankees got lucky and saw some career years that met his optimistic projections while exceeding most reasonable systems.

        Not to mention the rankings of the projection systems’ “success” depends heavily on how you define a “win.”

        The issue is the methodology of CAIRO is kind of BS. He’s just taking the Marcels, changing the league, park and aging adjustments in ways that make the Yankees look better and then making eyeball adjustments based on his own opinions. Then he runs them through Diamond Mind. That’s blurring the line between “projection system” and “guessing.”

        Vote -1 Vote +1

      • Sam says:

        Wow, I know Red Sox fans are generally an intelligent bunch, but some of these guys take their Yankee hatred way too seriously, and project that hatred towards anything remotely resembling a Yankee name.

        As far as “almost entirely objective” part goes, you have no understanding of SG’s (or Larry M’s) dry sense of humor. After all, the site is called “Replacement Level,” while being one of the most level-headed Yankee-centric site. And the projection system is named after Miguel Cairo, who I am sure you will agree is a Yankee legend.

        Of course, SG came up with far better rebuttals to your baseless accusations than I could ever come up with, so this post is really redundant, but man, was that dumb!

        Vote -1 Vote +1

      • Joe R says:

        Wow, I know Red Sox fans are generally an intelligent bunch…

        We are? Did I miss this memo? I just got out of the Red Sox fan convention, the first three items on the agenda is to cut JD Drew, move Ellsbury back to CF, and trade Kelly, Westmoreland, Beltre, Anderson, Buchholz, Bowden, Iglesias, and Gibson for Adrian Gonzalez.

        Vote -1 Vote +1

    • Mike K says:

      Interesting that you quoted it yourself but don’t seem to understand it. It is an “objective” system. The system you describe – which is not CAIRO – is a “subjective” system.

      Vote -1 Vote +1

  10. SG says:

    CAIRO just takes the Marcels and changes the underlying assumptions and components in a bunch of ways that make the Yankees look better.

    a) Can you show me any evidence that this is even close to true?
    b) What would be the benefit of rigging a projection system to make a specific team look better?

    This is seriously one of the stupidest things I’ve ever read.

    Vote -1 Vote +1

    • alskor says:

      a) Youve explained your methodology before. You’ve tweaked the Marcels, have you not? You changed a number of factors, including the weights of years, etc… Do you manually change players’ component factors to fit what you think they should be where you disagree with the Marcels? What exactly do you differently than the FANS system, for instance?

      Did the adjustments you’ve made impact the Yankee projections negatively or positively?

      Point blank – have you taken the Marcel projections and then changed the underlying assumptions (ie weights) – and have those changes made the Yankee projections look better? That’s all Ive accused you of, and I think that’s 100% what’s happened.

      b) If I had to speculate I would say most people that do this are probably of the belief that most projection systems didn’t give their favorite team’s players fair treatment, punished them unjustly for age, etc… (these are perfectly reasonable standpoints, btw).

      I dont have a problem with what youre doing at all… its perfectly reasonable to do that and an interesting exercise, especially for Yankee fans. Im just not sure it deserves billing alongside PECOTA, etc…

      Vote -1 Vote +1

      • Sky Kalkman says:

        If it’s historically just as accurate as PECOTA, why does it not deserve billing alongside PECOTA? CHONE is pretty much a glorified Marcel, too…

        Vote -1 Vote +1

      • Sky Kalkman says:

        Although, two years is hardly enough time to draw meaningful conclusions, I certainly admit.

        Vote -1 Vote +1

      • SG says:

        a) Youve explained your methodology before. You’ve tweaked the Marcels, have you not? You changed a number of factors, including the weights of years, etc… Do you manually change players’ component factors to fit what you think they should be where you disagree with the Marcels?

        Yes. Instead of using three years, I use four. Instead of ignoring minor league performance, I include MLEs. I use park factors, Marcel does not. Marcel regresses everyone towards league average, I regress towards a mean that includes some league average, age, and position. I use the same aging factors that they use, and I calculate playing time as a weighted average of the prior four seasons.

        There is nowhere that I make any ‘manual adjustments’, except in the obvious case of playing time issues.

        Did the adjustments you’ve made impact the Yankee projections negatively or positively?

        Without checking, my guess is they have impacted every team in some way because I think Marcel regresses more heavily than I do. Might it impact the Yankees positively? Possibly, but I don’t think it has, because I compare my projections with CHONE, Hardball Times, Marcel, PECOTA and ZiPS for the individual players during my pre-season projection reviews and they’re generally right in the middle of the pack.

        Point blank – have you taken the Marcel projections and then changed the underlying assumptions (ie weights) – and have those changes made the Yankee projections look better? That’s all Ive accused you of, and I think that’s 100% what’s happened.

        Well, no. You’re accusing me of being intellectually dishonest and posting something that I know is blatantly false to ‘make the Yankees look better’, even though I still don’t really understand what benefit that would serve. Would it make the Yankees perform better in real life? If I want to be taken seriously, doing something like that would make it a lot harder.

        I dont have a problem with what youre doing at all… its perfectly reasonable to do that and an interesting exercise, especially for Yankee fans. Im just not sure it deserves billing alongside PECOTA, etc…

        That’s certainly your opinion, and you’re entitled to it. I don’t have a problem with anyone who completely ignores CAIRO. I just don’t think making the accusations you’re making without any evidence of it or without an explanation of the motive behind it is fair, but you can say and think whatever you like.

        Vote -1 Vote +1

    • Sky Kalkman says:

      “This is seriously one of the stupidest things I’ve ever read.”

      Maybe. But that’s only your objective opinion.

      Vote -1 Vote +1

    • Dirty Water says:

      “What would be the benefit of rigging a projection system to make a specific team look better?”

      You can’t be serious. This ‘system’ was created by a Yankees fan, right? Of course it will be biased towards the Yankees. Otherwise, why create another redundant projection system.

      My guess is that whoever created it, became pissed last spring about other systems rating his Yankees low, so he went about ‘fixing’ that.

      Vote -1 Vote +1

      • Sam says:

        Just the same way CHONE is created to be “biased” towards the Angels, because Sean Smith happens to be an Angels fan? Or the Marcel system is now being “fixed” so that the Mariners come out on top, because Tango is being employed by the Mariners front office?

        Or maybe you just don’t understand the concept of personal integrity that a person puts on line when he/she puts up a projection system to be reviewed by everybody, goes at great lengths to explain the mechanisms of the system. You only understand the concept of fan bias. Perhaps cleaning up the dirty water may make a few things more transparent.

        I am rethinking my allegation of the Red Sox fan base as generally being intelligent.

        Vote -1 Vote +1

      • Dirty Water says:

        Generally, Seattle and LAA fans are not pretentious blo-hards. So even though I’m not familiar with the two who created CHONE and MARCEL, I will assume their work is legit. Something I will not assume about a Yankees fan.

        You indicated above that you use 4 years instead of the more common 3. Would your doing that have anything to do with the majority of your roster being pretty damn old, ie, in regression?

        Vote -1 Vote +1

      • Kevin S. says:

        That’s funny, you accusing someone else of being a pretentious blowhard. Ah, irony.

        Your position is not informed by any knowledge of the creator of the system, despite the fact that he’s been vouched for by other non-Yankee fans here. It’s informed by your unbridled hatred of anything related to the Yankees, and like in the past, it’s going to get the neutral crowd calling you out because you can’t discuss them rationally. Considering he applies an aging factor to the mean he regresses players to (something Marcel doesn’t), I’d say it’s unlikely he added the fourth year (which, in a weighted mean, is only going to have a small impact) to making older Yankees look better.

        Vote -1 Vote +1

      • Dirty Water says:

        Dude, I don’t need to understand the system to recognize that one which projects his Yankees with 102 wins is just wrong, and biased.

        What is the weighting system of those 4 years. I’d like to know.

        Vote -1 Vote +1

      • Sam says:

        You indicated above that you use 4 years instead of the more common 3. Would your doing that have anything to do with the majority of your roster being pretty damn old, ie, in regression?

        I am sure SG can answer that, but I think he believes that the more information you have to inform your projection, the better off you are. Also, his weighting scheme is more biased towards the recent past, with the last year receiving at least 40 percent of the weights, and the last year receiving at most 10 percent of the weight, as is true with any memory losing time-series process. So the fourth year is there to make more informed predictions about players who have at least 4 years of MLB history. I believe you could go five if you want to, or 6, but you also have to think about whether or not adding an extra year benefits you much in terms of being able to achieve better precision.

        And finally, really, look at CAIRO’s performance last year. It is possible it was a fluke, but it was based on being able to predict all 30 team’s performances over the year, not just one team.

        SG thought he is adding value through creating a projection system that, according to him, is a better estimator of player performances, and the results back him up in a limited sample. As he himself has mentioned, you are free to reject is methodology and results. But at least write something coherent and logical in attempting to do so.

        Vote -1 Vote +1

      • TCQ says:

        “Generally, Seattle and LAA fans are not pretentious blo-hards. So even though I’m not familiar with the two who created CHONE and MARCEL, I will assume their work is legit. Something I will not assume about a Yankees fan.”

        So basically what you’re saying is that you won’t believe anything a Yankees fan says.

        It really shouldn’t even have to be explained to you how stupid this is.

        Vote -1 Vote +1

      • xeifrank says:

        Guys, stop feeding the troll.

        Vote -1 Vote +1

      • Dirty Water says:

        Well, I went looking everywhere on that site, Sam, and could not find anything that explains how such odd numbers were derived. And really, any system that projects the obviously subpar defense and thin pitching staff of the Yankees to allow less than 700 runs, really, really, really, needs explanation.

        I almost lost my lunch after reading that one.

        Xei, piss off

        Vote -1 Vote +1

      • Kevin S. says:

        What needs explanation is your characteristics of the Yankee defense and pitching staff. Say what you will about Brett Gardner’s offense, he’s going to provide a plus-plus glove if he stays in left, Granderson’s above-average in center, and Swisher’s been consistently average in RF. As for the infield defense, the glass half full look says Jeter’s improvements are here to stay, A-Rod’s bad defense was a lingering side effect of the surgery (and he did seem to gain mobility as the season went along), UZR does a bad job with 1B defense and Cano keeps his head in the game. Half empty says Jeter’s defensive surge was an anomaly, Teix is just average at first, Cano sucks and A-Rod can’t handle third. I’d probably eyeball it at average to a little below average, although of course I’m guessing you went with the most pessimistic views there. Overall, I’d say it’s a middle of the pack defense, which is just fine behind a staff that will generate a lot of strikeouts. By the way, I’m not exactly sure how having one of the top five pitchers in the AL, two slightly above-average guys good for 200 innings, and a pitcher who posted three straight ~5 win seasons in a hitters park in the AL… before having a career year constitutes a thin rotation. On most teams, Joba and Hughes would be penciled in as 3rd and 4th starters. For the Yankees, one has to go to the bullpen. I guess the Yankees didn’t sign enough 42 y.o. reclamation projects in the offseason for your liking, though.

        Vote -1 Vote +1

      • Dirty Water says:

        Based on what I saw last year, Gardner will not hold the position; neither will Winn. Maybe that’s where SG got his 102 wins – he’s assuming the Yankees acquire Pujols for LF

        But I was actually referring to the Yankees staff career GB% of 46%, and Arod, Jeter and Cano somehow stopping those balls.

        Under 700 runs is a ridiculous claim with such a porous INF. Their OF is OK, though, I guess.

        Vote -1 Vote +1

      • Sam says:

        Well, I went looking everywhere on that site, Sam, and could not find anything that explains how such odd numbers were derived.

        The site is in the process of being updated, so wait for it. SG does this every year, multiple times, sometimes inside the season. When he does these iterations, keep an eye for it, and if you have constructive suggestions, SG will accommodate them, I know from experience.

        Vote -1 Vote +1

      • Brian Cartwright says:

        Oliver says the Pirates still suck. Nothing much I can do to tweak the system to make it say otherwise.

        Vote -1 Vote +1

      • B N says:

        Actually, I just finished my new projection system and it is showing the Pirates finishing somewhere around 102.55555556 Wins, but being knocked out of the ALCS in 2. It may still need some tweaking, but at face I’m pretty sure it’s legit. :)

        Vote -1 Vote +1

      • LKH says:

        “And really, any system that projects the obviously subpar defense and thin pitching staff of the Yankees to allow less than 700 runs, really, really, really, needs explanation.”

        And they eventually only gave up 693 runs.

        So you were saying, Dirty Water?

        Vote -1 Vote +1

  11. SG says:

    And 2009 was year 2 of CAIRO, not year 1.

    Vote -1 Vote +1

  12. Ken says:

    You suggest that the difference between prediction and projection should affect the way projections are discussed. While this is obviously true, it should also affect the way they are judged. Assessing the quality of a projection by measuring its divergence from outcomes, which seems to be universally accepted, is only reasonable if these should be read as predictions.

    Vote -1 Vote +1

    • B N says:

      Of course, if you don’t measure the quality against outcomes…. I’m not exactly where that leaves you with measuring it at all. Unless you have a counterfactual universe generator that I don’t know about. And if you do, I want in, Doc Brown.

      As near as I can tell, projections are all stated in a form of a mean, median, or mode type statement. Sort of a “this is the most likely thing.” Firstly, unless you’re assuming you’re using medians, you’re not even minimizing your variance so you’re really not projecting something that is likely to have a low distance from outcomes. I don’t think all these systems are designed to be compared to outcomes usefully. More so, I don’t know if their distance from outcomes would make them any more “useful” to people who look at them.

      For example, take J.D. Drew. I could make two projections for the number of games he’ll play next year, one based upon what I think his median outcome would be and one based on what I think is his most likely outcome. If I’m good at this and have good info, my median analysis is likely to have the least divergence (let’s say 100 games, for example). With that said, I might rather have a projection that assumes his most likely number of games (let’s say 120 games, for example). So my median-value projection is weighted down by the possibility that he trips on the plane to spring training and misses the whole season. I can think of reasons (see: fantasy baseball) that I would rather see the most likely outcome rather than one that attempts to actually balance the over/under variance.

      Basically, projection systems could only be gauged based upon what they’re actually trying to project (which still seems somewhat unclear to me on some of these…). And moreover, they’re bound to be unduly affected by qualitatively different circumstances and external events.

      In my opinion, there’s not a lot of sense trying to evaluate projection systems except for their utility on whatever you’re using them for. If they help you make decisions, giving you some information, they’re helpful. But they’re definitely not predictions because they’re not taking into account much of the important information. They’re also stating just one potential point in a huge possible distribution. Just because that “one point” is a guess to the performance of every player doesn’t mean its any more than one point. Plus, since player performances covary, an individual projection can’t ever be stated as a reliable prediction (let alone evaluated as one). Even trying to figure out how well a system performs on any given year is a waste. You have a divergence, but no confidence. Just because a system was good for one year across, or even 10, that’s still a small sample size.

      Also, I’m no statistician but I’m pretty sure if you wanted to be able to evaluate different prediction generators- you might be best off having one that estimates a distribution of the counterfactuals of the season, then checks the likelihood that the observed season falls into that distribution. Just saying… doable, but probably more trouble than it’s worth.

      Vote -1 Vote +1

  13. Circlechange11 says:

    Ken I agree.. The ‘Thumbs Down Legion’ is all over comments suggesting that projections are by application predictions.

    We basically view them as statistically-based predictions, use them as predictions to make choices and evaluations of the future, and judge them based on their accuracy of future events.

    Regardless of whether we refer to them as predictions or not we treat them as such. Certainly labels and usage being incongruent is not a new situation.

    If they’re NOT a type of prediction, then why do we care about them? And talk about them? And refer to them?

    Vote -1 Vote +1

    • Nathaniel Dawson says:

      Perhaps you treat them as predictions — but I’m pretty sure most other people that are knowledgeable about the intent and purpose of them don’t. I, for one, don’t. They’re just a best guess at this point what the most likely outcomes are. Everyone that’s been a fan of baseball for much time knows that players and teams have up and down years, and the end result can be very different from what you would expect. This is just a look at what might be the “true talent level” for a team as a whole and how that talent level stacks up against other teams. And a rough estimate of their playoff chances, as well. How those teams and players are going to produce over the season is something we can’t know.

      Vote -1 Vote +1

      • vivaelpujols says:

        Well they are predictions. It is a semantic issue. Projections try to nail down true talent level, essentially, which is basically how well you expect that player to do going forward. Predictions try to predict how well that player will do going forward.

        Projections are predictions. They predict the most likely outcome.

        Vote -1 Vote +1

      • CircleChange11 says:

        They’re just a best guess at this point what the most likely outcomes are.

        Well, what would your definition of a “prediction” be in terms of what we are discussing here (i.e., full stat lines for every major league player)? [Serious question, asked politely]

        How would those “predictions” differ from the “projections”. [Another serious question]

        Others have been much more polite than I am in the mood to be, because when I read the original article it seemed like, just another reminder to all of us idiots and morons who has achieved Jedi Master understanding of baseball projections and who has not.

        We have a thumbs up and thumbs down icon here at FG. If we had a “jack off” hand motion icon, I would have voted that for the original article because it read, to me, as just high-handed, down the nose, completely unnecessary commentary on something that matters little in every day stat applications.

        Occasionally my dad will say some stuff that ispretty clever. He once taught me that if you have to go around telling everyone how smart or good you are, then it simply isn’t obvious enough … and you should spend your time on practicing and improving performance instead of boasting or preaching.

        Now, if there was an “actual” difference between projections and predictions, in terms of how they’d be calculated/figured/etc and how they would be used by stats fans, then it would be obvious and we wouldn’t have to “nickel and dime” the discussion to death.

        Now, if someone could explain to me how differently statistical predictions would appear, be calculated, figured, derived, etc from baseball projections … then maybe I would see the glaring distinction and increase my intelligence and understanding.

        None of this is intended to be as harsh as it will likely read to some, but at some point we’ve gotta get past all this “who’s smartest” type crap, and either do an excellent job teaching the differences or just not waste time pointing it out.

        Vote -1 Vote +1

      • Colin Wyers says:

        “Projections are predictions. They predict the most likely outcome.”

        No, they don’t. They predict the outcome with the least expected error.

        Vote -1 Vote +1

  14. Nate Marcum says:

    I am going out on a limb and guessing that you are from Missouri? I could be wrong, but if you happen to be from here, beers are in order for great articles. I only figure that because of the use of HW-40.

    Vote -1 Vote +1

  15. Joe R says:

    And my fellow Red Sox fans getting uppity over a projection system made by a Yankee fan kind of reminds me of the fan uproar in NY when a group of statisticians rated Jeter as the worst defensive SS and the place they presented their analysis just happened to be Boston.

    OMG CANNOT TRUST THIS.

    God, my childhood unmitigated hate towards the Yankees has forever been replaced by an unmitigating hate for stupidity. How depressing.

    Vote -1 Vote +1

    • Kevin S. says:

      Yeah, but hatred of stupidity can make strange bedfellows. The beer’s on me next time you hit Lower Manhattan, Joe.

      Vote -1 Vote +1

  16. Dave S. says:

    SG rigged his projections! SG rigged his projections!

    The ignorance of such posters would be hilarious if it didn’t introduce a toxic element to the conversation – not unlike a bunch of Intelligent Design believers at a biology conference. Without even reading the text to go along with it – SG suspects that the Yanks are worse than their projection, the Sox and Rays better, is suspicious of the Blue Jays, fills it with the caveat that it’s a late January projection – you would insist that SG has fixed his projections to make the Yankees look better. Setting aside the accusation of intellectual dishonesty even for the moment, what real benefit would it be for a professed Yankee sabr-man to overstate the record for his team? If one wasn’t committed to simply understanding the game, and one was in fact trying to rig a projection, one would prefer to UNDER project his team’s performance, no? Such fanboy behavior would result in the ability to tout the “intangibles” of his team’s performance at the end of the year, or to say such things as “you know, I thought they would be good, but not THIS good. They have that certain magic something, you know?” But you guys see some early return projection numbers, filled with caveats duly noted by the author, and accuse without any real understanding said author with data tampering? It’s a joke. You’re idiots who have missed on the point on so many levels you might as well go back to touting batting average and pitchers’ wins.

    Vote -1 Vote +1

    • Dirty Water says:

      Who cares if SG rigged his projections! Who cares if SG rigged his projections!

      I care that FG gave it equal billing to CHONE. ZiPS. MARCEL and PECOTA, when even, as you point out, its programmer admits its many flaws.

      Vote -1 Vote +1

      • What's wrong with you says:

        Christ.

        Vote -1 Vote +1

      • J. Cross says:

        This is pretty ridiculous. CAIRO clearly more “sophisticated” than Marcel given his description; all of these systems have flaws. And, btw, *all* good projections systems including the mighty PECOTA are in one way or another tweaks of the basic marcel idea. Those tweaks might not add a ton of predictive power but I think they’re pretty interesting in and of themselves.

        Vote -1 Vote +1

      • vivaelpujols says:

        CAIRO is definitely better than Marcel’s, and I suspect it’s very similar to CHONE and ZIPS in terms of methodology (although I think ZIPS is probably better in terms of regressing individual components, and CHONE at aging). PECOTA is the most unique and complex, but that doesn’t really manifest itself in the accuracy testings.

        Vote -1 Vote +1

      • Sky Kalkman says:

        I think this is a great point, from viva:

        “PECOTA is the most unique and complex, but that doesn’t really manifest itself in the accuracy testings.”

        I have a lot of respect for the thinking and concepts that went into creating PECOTA. Conceptually, it’s really neat.

        But as of now, it’s not any better than CHONE/ZiPS/CAIRO/Oliver. Maybe PECOTA is the closest of the bunch to taking the next step. Or maybe if PECOTA combined its approach with some of the things that other systems do, it would pull ahead. But as of right now, in terms of accuracy it’s not the leader of the pack.

        Vote -1 Vote +1

  17. Bruce says:

    You’re missing the point. What would it accomplish to have him rig projections?

    Vote -1 Vote +1

  18. Mustard & Brown says:

    I am near positive I will get slammed for saying the following…but..

    1) I don’t post much here, but read EVERY day…I think these type threads are what keep me going back to BPro and BTF rather than camping out here. The feeding the troll at FanGraphs is constant…and it gets old. At times people in these threads just can’t let the opportunity to tell another they are an idiot pass. Whatever happened to ignoring people when they reach a certain point?

    2) (…and I’m sure this will be regarded as fallacy of submitting to authority)…as an academic statistics professor, I can assure you that the issue of prediction vs. projection IS semantics. I understand fully the point Dave is trying to make, –the INTERPRETATION of these pre-season numbers is the issue. But, CircleChange is correct when he states that if we ask for predictions of what players of teams will do at this point, we will go thru the same process as we do for projections. This is because we cannot (yet, accurately) statistically model the changes in season that Dave is talking about.
    The bottom line is that all predictions/projections/estimates are meaningless without margins of error. This is why it is important to hear what SG is saying about a standard deviation being as high as 10 games, etc.

    3) I would agree with those that state CAIRO should not be set aside PECOTA, ZiPS and CHONE. But not because a Yankee fan runs it. That is really a silly reason not to give something its due. The Padre die-hard in me should care nothing that SG is a Yankee Fan or Sean runs an Angel Blog or Dan at BTF lives near Baltimore. C’mon. CAIRO is just a “younger” method.. in due time, it may very well belong among those more established. I for one would like to see CAIRO more “available”. It seems to a little less “out there” for the eyes to see than the others, but I expect that will change.

    I sincerely apologize if I have popped my head up from my usual “lurking” mode and offended the regular posters.

    Vote -1 Vote +1

    • SG says:

      I would agree with those that state CAIRO should not be set aside PECOTA, ZiPS and CHONE. But not because a Yankee fan runs it. That is really a silly reason not to give something its due. The Padre die-hard in me should care nothing that SG is a Yankee Fan or Sean runs an Angel Blog or Dan at BTF lives near Baltimore. C’mon. CAIRO is just a “younger” method.. in due time, it may very well belong among those more established. I for one would like to see CAIRO more “available”. It seems to a little less “out there” for the eyes to see than the others, but I expect that will change.

      I think this is completely fair. After all, we know the importance of sample size.

      Vote -1 Vote +1

    • wobatus says:

      I am so glad you posted this.

      Vote -1 Vote +1

    • B N says:

      Firstly, as you have just noted that you don’t post very much- I want to say that I found your post very interesting and well thought out and you should probably post more.

      Secondly, I still disagree with some of what you’re saying… particularly: “CircleChange is correct when he states that if we ask for predictions of what players of teams will do at this point, we will go thru the same process as we do for projections.”

      While that may be true, that simply indicates that we’re just slapping the label of “prediction” on a projection. However, I don’t think that makes them the same thing, any more than if I took a 2-D square and labeled it a cube.

      By my understanding of how projections seem to be defined versus predictions, a projection takes prior states from the system and attempts to project later states of the system. You’re treating a system as essentially endogenous, maybe with some error sources. So projection is like Projection = f(System History)

      I would contend that a prediction is more like: Prediction = g(System History, Exogenous Data, Possible Changes to System). A prediction does not necessarily have to be bound by the prior performance of the system in question. It accepts the possibility of exogenous effects or qualitative shifts in dynamics. In my view, a prediction lifts the assumption that past dynamics remain valid (but may still apply such principles to the degree they are useful). This allows assumptions about dynamics or influences that would not make sense in a pure projection.

      For example, if it was announced that all players in the MLB would now play with aluminum bats rather than wood, and superballs instead of cork/- a projection is not going to handle that. With that said, a prediction could still be made. By determining the impact of aluminum bats vs wood bats in other contexts, analogical inferences might be made (ex. aluminum causes 10% longer distance on FB and LD). If I make that adjustment to my estimates of ball distances, I’m not doing a projection.

      I’d say that considering the possible impact of new exogenous influences or dynamics is a pretty major shift in process. Even if you end up saying “Oh, well, no major anticipated dynamics/exogenous factor changes this year”- considering that is an extra process step. Even if the outcome is the same, the process isn’t.

      Of course, you can play games with what is a prediction and what is a projection by changing the scope of how you define your system versus your externalities- but that’s a whole other issue.

      The moral of the story is that I think there are very important differences between predictions and projections with regards to the process (though not necessarily the outcome). There is also a difference in that predictions are implied to be more accurate than “forecasts” or “projections” but that has always seemed pretty subjective to me.

      Moreover, the point that Cameron seemed to be making appeared to be that we’re not considering the factors exogenous to the system- even though we know it will happen and will have a driving force. That the projection systems implicitly hold factors constant by omission, because they’re not even addressed as part of the random error. And if we were predicting, we’d have to address in some way why we just hold those constant (i.e. everyone staying on the same teams, etc) as part of the process.

      Vote -1 Vote +1

      • B N says:

        PS I do not actually know the statistics on batted ball distance with different type bats. 10% is a pure fabrication. Do not attempt to rely on this when selecting bats in your company baseball/softball league.

        Vote -1 Vote +1

  19. murr says:

    I’m going to assume that they’re not trolls, but humans capable of rational thought. If so, they owe SG an apology for attacking his intellectual integrity and continuing to do so even after their accusations were refuted. Anyone who knows SG also knows his intellectual honesty is unassailable.

    Vote -1 Vote +1

  20. quincy0191 says:

    Fan Approximations for Next Season

    Vote -1 Vote +1

  21. MD687 says:

    Beautiful post and great examples.

    Vote -1 Vote +1

  22. alskor says:

    Alex Rodriguez
    Marcel: 527 mPAs, 31 HRs
    CAIRO: 605 PAs, 37 HRs

    Let me get this straight, you used a “a weighted average of the prior four seasons” and got 605 PAs for ARod? What kind of weighting did you use here…?? Yet, you claim you didn’t manually tweak these…?

    Mark Teixeira
    Marcel: 622 PAs, 31 HRs
    CAIRO: 670 PAs(!), 34 HRs

    Kevin Youkilis
    Marcel: 556 PAs, 22 HRs, .390 wOBA
    CAIRO: 615 PAs, 22 HRs, .383 wOBA

    Dustin Pedroia
    Marcel: 630 PAs, 14 HRs, .370 wOBA
    CAIRO: 669 PAs, 14 HRs, .357 wOBA

    AJ Burnett
    Marcel: 12 Ws, 185 IP, 176 H, 177 K, 80 BB, 22 HR, 4.26 ERA
    CAIRO: 13 Ws, 187 IP, 172 H, 181 K, 77 BBs, 19 HRs, 4.02 ERA

    Vote -1 Vote +1

    • Sam says:

      Since you just won’t give up your idiocy after being thoroughly claimed, here it is for you:

      Alex Rodriguez (4/3/2/1 weights over last 4 years):

      (535*4+594*3+708*2+674)/10=605.3

      Now please be human enough to apologize to SG. Of course, I know that is asking for too much.

      Vote -1 Vote +1

    • vivaelpujols says:

      If you can show, that, systematically, CAIRO projections are over-projecting Yankees and under-projecting Red Sox or Rays, compared to Marcel’s, it might be more believable. Cherry picking 5 players doesn’t do much to prove your point.

      Vote -1 Vote +1

      • alskor says:

        Take a look at the spreadsheets.

        Vote -1 Vote +1

      • vivaelpujols says:

        Yeah, it’s not really my job to prove your point. I’m going under the assumption that CAIRO projections are unbiased, based off of SG’s clear explanation of the methodology and the fact that it has been one of the more accurate projections over the past 2 years. You’re the one advancing the alternate theory that his projections are biased.

        Vote -1 Vote +1

    • SG says:

      I am not sure why I’m responding to you again, but this is the last time.

      Player PA BRAR RS WAR
      Derek Jeter 697 32 -4 2.8
      Nick Johnson 441 10 0 1.0
      Mark Teixeira 670 34 3 3.8
      Alex Rodriguez 605 44 -4 4.0
      Curtis Granderson 676 28 6 3.4
      Jorge Posada 426 21 -5 1.6
      Robinson Cano 642 31 -1 3.0
      Nick Swisher 619 18 0 1.8
      Brett Gardner 435 6 2 0.8
      Total 5211 225 -3 22.2

      Player PA BRAR RS WAR
      Jacoby Ellsbury 628 17 2 1.9
      Dustin Pedroia 669 33 9 4.2
      Victor Martinez 670 38 3 4.1
      Kevin Youkilis 615 28 4 3.2
      David Ortiz 604 20 0 2.0
      Mike Cameron 605 18 5 2.4
      J.D. Drew 529 22 2 2.4
      Adrian Beltre 575 8 8 1.6
      Marco Scutaro 562 17 3 2.0
      Total 5457 202 36 23.8

      Those are the values for plate appearances (PAs), batting runs above replacement(BRAR), projected runs saved defensively(RS), and wins above replacment (WAR) for the two starting lineups according to CAIRO. Apparently I need to tweak my bias weights more.

      Vote -1 Vote +1

      • alskor says:

        Ok, humor me one more time now… if your projections for the position players come out like that… how are you coming up with a 10 win difference between the two teams…?

        Vote -1 Vote +1

      • Dirty Water says:

        He has the respective lineups WAR about right. I could argue a few of them but on the whole they appear OK. But he has the Yankees saving 58 additional runs next year (yet the Sox only save an additional 34. WTF? Sox RA should realize a massive improvement, by any projection system) – which has to come from the new Gardner/WInn, Grandy OF, and Vasquez. But no other system has the Yankees giving up so few runs, and as an observer, I can see why:

        RA:
        -Granderson Damon
        -Arod, Jeter and Posada of 2009 > Arod, Jeter and Posada of 2010, ie, one year older.
        -Vasquez is surely better than the various starters he’ll replace from 2009 but I doubt he’ll be much better than how those starters actually performed, a most fortunate year for Yankee starters.

        Vote -1 Vote +1

      • Dirty Water says:

        sheesh

        RA:
        – 2009 Melky/Gardner is better than Granderson
        – Gardner/Winn is better than 2009 Damon
        – Arod, Jeter and Posada of 2009 > Arod, Jeter and Posada of 2010, ie, one year older.
        – Vasquez is surely better than the various starters he’ll replace from 2009 but I doubt he’ll be much better than how those starters actually performed, a most fortunate year for Yankee starters.

        Vote -1 Vote +1

      • Sam says:

        Have you considered strength of schedule? I think that would play an important part in final standings, and from what I understand, the Sox and the Rays (remember them?) play a much tougher schedule this year. So when Diamond Mind runs them, it probably has the Yankees giving up fewer runs compared to RS, even though John Kruk knows Red Sox have approximately 12 aces.

        Plus, this really is version 0.1.1 or something like that with possible coding errors in them and based on 100 seasons (usually SG does 1000 seasons for his final projections). Way too early to get worked up on anything. On top of that, the margin of error is 10 wins or something. Yankees could end up winning 92 games or fewer (as I am sure happened in some of the simulations when Kei Igawa was called upon to pitch), and the Sox could end up winning 102 games or more. All this tells is based on about 100 seasons worth of data, the average season turned out to be 100 plus wins for Yankees and 93 wins for the Sox. So take a xanax and chill out.

        Vote -1 Vote +1

      • Red Sox Fanboy says:

        RA:
        – 2009 Melky/Gardner is better than Granderson
        – Gardner/Winn is better than 2009 Damon
        – Arod, Jeter and Posada of 2009 > Arod, Jeter and Posada of 2010, ie, one year older.
        – Vasquez is surely better than the various starters he’ll replace from 2009 but I doubt he’ll be much better than how those starters actually performed, a most fortunate year for Yankee starters.

        Tough to argue with such a cogent, well-thought out analysis like this. Vazquez is beter than the various starters he’ll replace, but I also doubt he’ll do better than how they performed. I see him matching Wang and Mitre’s 8+ ERA at best.

        Vote -1 Vote +1

  23. philosofool says:

    “I’m traveling east on I-40. This map expects me to drive into the Atlantic Ocean.”

    Actually, having lived in Wilmington, NC, for two years, as I recall, I-40 becomes a big street (College Road) that runs through Wilmington, to Myrtle Beach, through Kure Beach, and then right into the beach itself. So technically, I think it does expect that.

    Nothing deep is meant by this comment. I’m just reflecting on two years of first-hand geography lessons on the worst laid-out city I’ve ever lived in.

    Vote -1 Vote +1

  24. Bucksky619 says:

    You are making a prediction. Saying that such and such will happen if nothing changes is still making a prediction. Unlike a map which is based on indisputable information, a baseball projection system can only make a guess based on whatever information it uses. Maybe it would be helpful if you more clearly defined what would be considered a change in course. You mentioned how a team might get better if it acquired a new player but what else is considered? If a player has a career year and blows away his avg. annual production is that considered a change in course? Injuries? Does a top prospect giving up baseball for the priesthood impact projections?

    Vote -1 Vote +1

    • B N says:

      It’s only a prediction if you took an extra step to decide if “nothing changes” or not. If your entire methodology starts with the assumption “and nothing else changes the system” then you’re doing a projection. If you define your system, then define as much as you can of outside factors, then say “Well, for this analysis we’re just going to hold all these constant” then you’re doing a prediction. In fact, you’re doing a prediction that utilizes a projection to get its estimates. But you didn’t just start your analysis based on the “nothing changes” assumption.

      Vote -1 Vote +1

  25. murr says:

    8.5 win difference, not 10. The cherry picking is followed by another distortion, and not by an apology for besmirching SG’s intellectual integrity, despite his repeated patient explanations.

    Vote -1 Vote +1

  26. pounded clown says:

    Screw CAIRO, just call NORAD and have them boot up the WOPR

    Vote -1 Vote +1

  27. wobatus says:

    If it was close to on-target, it was a prediction; if it misses by a mile, it was just a projection.

    Vote -1 Vote +1

    • B N says:

      That’s funny. I always thought if it was close on target, it was a smart bomb and if it missed by a mile it was a Mars rover lander. But to each his own, I suppose.

      But yah, that’s a definition that people use too- but it’s a pretty junky one since then it’s all purely subjective and application specific.

      Vote -1 Vote +1

  28. pounded clown says:

    Statisticians might say “projections”; lay people might say “predictions”, while others whose bathroom literature includes the Turner Diaries and the Bible say
    “prophecy”

    Vote -1 Vote +1

  29. BD says:

    ” Vasquez is surely better than the various starters he’ll replace from 2009 but I doubt he’ll be much better than how those starters actually performed, a most fortunate year for Yankee starters.”

    I’m just cherry-picking the above-quoted remark in order to point out something pretty basic: A couple people here can’t seem to accept the possibility that ANY projection system that has the NYY winning 102 games could be legitimate. But the Yankees in fact won 103 games last year. You say you doubt Vasquez will be “much better” than his 2009 counterparts (not sure who you mean, btw), but why would he have to be better at all, in order to justify a projection of one FEWER win than the Yankees achieved last year?

    Vote -1 Vote +1

  30. BD says:

    As for the “prediction” vs. “projection” controversy, I think of it this way: A baseball projection system is essentially just a rating of the strength of a given roster, expressed in terms of estimated wins over a 162-game season. A projected 90-win team has a somewhat better aggregation of players than a projected 85-win team. But the projected 90-win team won’t necessarily finish higher in the standings than an 85-win team for a whole lot of reasons, which is why the play the games.

    An analogy might be auto racing. Someone can analyze the actual racecars and declare that driver x’s vehicle is one mile an hour faster (at top speed) than driver y’s. Logically, that implies x has an edge over y in an upcoming 200-mile race; but that edge, in and of itself, is nowhere near enough to support a confident prediction that x will actually beat y. There are just so many other factors that will come into play over the course of an entire race (acceleration, handling, driver skill, pit crews, latent mechanical problems, timing of yellow flags, accidents caused by other drivers affecting x and y differently, etc.).

    Vote -1 Vote +1

    • B N says:

      I don’t quite see how a prediction system has to be any different than what was described. As near as I can tell, you’re just talking about differences in the types of factors you’re willing to include in the system you’re looking at. That’s kind of like saying that if I used just RBI totals to estimate future RBI totals, that would be a projection- but if I included the rest of standard stats it wouldn’t be. But it would still be a projection.

      It would be a prediction if I had some sort of methdology for figuring out why I’m selecting the stats I’m using in the first place. Or if I had a step where I considered how rule changes might alter RBI totals.

      Vote -1 Vote +1

    • B N says:

      Sorry, didn’t quite finish. So that could be a prediction or a projection. It depends how you’re defining your system and what your options are for dealing with stuff outside of that definition.

      Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>