2012 Organizational Rankings: Methodology

On Monday, we’ll roll out the beginning of our annual Organizational Rankings series – the plan is to do three teams a day for 10 days, concluding with the top three teams on April 4th. Before we begin discussing each franchise, however, I thought it’d be helpful to explain some of the changes we’ve made to the system this year.

Last year, we made one key change that we’re carrying over, and will likely remain in place going forward: our authors were asked to grade individual inputs rather than overall organizational strength. Grading by components adds a layer of transparency that is important, as you can see exactly why a team ended up placing in the specific spot they land.

However, we’re upgrading the implementation of that grading scale this year. Last year, I asked all of our authors to assign letter grades to each organization in three categories: Present Talent, Financial Resources, and Baseball Operations, and then had the three guys on staff who specialized in prospect analysis grade them out in terms of Future Talent as well. We then converted the letter grades for those four variables into scores, assigned weights to the individual categories, and used the weighted averages to create a total score.

There were a few problems that arose by doing things that way, however, and we’re doing our best to fix those issues this year. Here’s how:

1. The grading scale has been changed from letter grades (A-F) to the 20-80 scale that is commonly used by baseball scouts. The 20-80 scale is actually a pretty good system, with 50 as a clearly defined average and each 10 point increment representing one standard deviation from the mean. We asked each of our writers to use this scale for their grading, and to grade in five point increments (50, 55, 60, etc…), and to ensure that their average overall score for each category came out very close to 50. With the letter grading scale, the overall average was simply too high, and this allowed us to easily adjust the grades if an author’s overall average came in too high. This scale allows more consistency between authors, and gives us a better representation of what we’re trying to communicate about a team in a certain area.

2. Present Talent and Future Talent have been more clearly defined, and will now be referred to as “2012 Outlook” and “2013+ Outlook”. That category is not just current prospects or recently graduated prospects, but also the expected future value of every player under team control beyond this season. We want to give teams credit not just for their prospects or young Major Leaguers, but also for the quality pieces they already have in place for the next several years. We also realized that only having our prospect writers grade out this category didn’t make sense and led to several unintended consequences, so everyone participated in the 2013+ Outlook category this year.

3. The weights placed on each category have been modified slightly, with the biggest change coming in the level of importance placed on a team’s baseball operations department. Last year, the weightings were 30/30/25/15, with Present Talent and Financial Resources carrying the most weight, then Baseball Operations, and Future Talent coming in as the least important variable. This year, the weightings are 35/35/15/15, so we essentially reduced the baseball ops importance by 40% and divided those points equally between 2012 Outlook and Financial Resources. This was done for a couple of reasons:

a. Baseball Operations is the most fluid part of any organization, and can be changed relatively easily. While a team cannot quickly overhaul its talent base or instantly begin generating new revenues, a front office can be turned over in short order and new processes put into place fairly quickly. The Astros are the perfect example of this, as a year ago they graded out as having the worst baseball operations staff in the league, but have since made significant strides to change the ways they make decisions. Since this category is so fluid, it doesn’t make sense to credit or penalize a team so strongly for something that isn’t static.

b. This is also the area we know the least about. We can measure player talent with some degree of confidence, and we can measure financial resources very easily, but without sitting in on meetings that we’re not privy to, it’s impossible to know exactly what kind of inputs are going into a team’s decision making. We can make some some educated guesses based on the kinds of transactions that a team completes, but we also don’t see any evidence of decisions that result in no transaction, and what we end up drawing conclusions based off is a very small part of what a baseball operations staff actually does. At this point, every team in baseball has analytical capabilities, and the differences in how they operate are mostly decided by how much emphasis is placed on those tools at the executive level. That’s just something we can’t really know from the outside, so our uncertainty should serve to make us less confident in our evaluations of this category, and as such, it should carry less weight.

Making these changes should help deal with a few of the issues that have arisen in the past. That doesn’t mean these rankings will be perfect, or that you’ll agree with all of them, but we hope that you’ll be able to see why the consensus of the group led to a team being positioned in a specific spot within each category.

Finally, we’re auctioning off the #6 spot on the list*. After the Mariners in 2010 and the Twins last year, we now see that placing sixth on this list is a near guarantee of epic failure, so if there’s a team that you’d like to see crash and burn in 2012, simply send us a large sum of money along with the franchise that you’d like to sabotage. The largest donor of the day will get to ruin the organization of his choice.

*Okay, we’re not really doing this. But we thought about it. Maybe next year.




Print This Post



Dave is a co-founder of USSMariner.com and contributes to the Wall Street Journal.


44 Responses to “2012 Organizational Rankings: Methodology”

You can follow any responses to this entry through the RSS 2.0 feed.
  1. jjdouglas says:

    #6 for the Yankees please.

    +44 Vote -1 Vote +1

  2. Mad Dan says:

    Looking forward to this.

    In the future, I think the present talent component could be easily computed using your estimates from the positional rankings run a few weeks ago. Simply total the projected WAR for each team, and you’re set – more analytical and less subjective.

    Vote -1 Vote +1

    • Baltar says:

      That exercise was clearly a quick and dirty–not expected to be accurate or precise, just informative food for thought. It was also very dependent on pure guesses about what management will do with the playing time and positions of many of the players.

      Vote -1 Vote +1

  3. Drakos says:

    So do these rankings include the fiasco going on with Moorad and the Padres right now? Cause that’s making things interesting as a fan.

    Vote -1 Vote +1

    • Dave Cameron says:

      The baseball operations category isn’t just the GM and his staff. It includes essentially anyone on staff from the owner to the trainers who might have an impact on the ability of the club to acquire and develop talent.

      Vote -1 Vote +1

      • Drakos says:

        I was thinking more along the lines of Financial Resources considering that the person leading the ownership group attempting to take control of the Padres is no longer the CEO and doesn’t have an office at the ballpark anymore. And this news came out last night.

        Vote -1 Vote +1

      • Dave Cameron says:

        Theoretically, that’s part of the financial resources ranking. We completed these before news of the Moorad/Moores split came out, though, so they won’t be reflecting the latest information.

        Vote -1 Vote +1

  4. Martin says:

    What is the direction given to the people doing the rankings on “Financial Resources?” I remember from last year there was some criticism in the comments about how certain teams were placed behind other teams in this category.

    Vote -1 Vote +1

    • Baltar says:

      There was some criticism about everything, and Financial Resources was probably the least controversial.
      This exercise is supposed to be informational and fun, not a doctoral thesis.

      Vote -1 Vote +1

  5. t ball says:

    I applaud your willingness to continually reexamine process, consistent with the overarching goal of saber thinking.

    Vote -1 Vote +1

  6. Sky says:

    Where do salaries come into play? Better players count towards Talent but large contracts hurt Resources? Or do performance and salary combine into Talent?

    Vote -1 Vote +1

    • Dave Cameron says:

      They’re in financial resources, which is essentially a combination market-based revenues (how large is the metro area, what kind of TV contract do they have, do they own their own station, etc…), ownerships access to capital (the Mets having the Wilpons drags them down from where an NY-based team should be), and future contract obligations (how much of their expected future payrolls is already tied up in commitments).

      The two outlook categories are strictly based on talent. Salaries are not considered there.

      Vote -1 Vote +1

  7. therood says:

    …so the White Sox will be at the bottom of another ranking then. I am [prepared for] disappoint.

    Vote -1 Vote +1

  8. Snapper says:

    I look forward to this exercise, it’s a lot of fun.

    So, the no brainers are probably #30 Houston, #1 Yankees?

    Vote -1 Vote +1

    • Ludwig von Koopa says:

      Texas probably has a shot at #1, I’d say. Despite the Yankees being the Yankees I’d imagine they’d get some kind of malus for their large future salary commitments. But it has to be one of them, I’d say.

      Vote -1 Vote +1

      • Snapper says:

        Even with the $75M committed to ARod/CC/Tex, their remaining payroll is probably at least equal to Tex.

        And those guys should be producing significant value, especially CC, even if ARod and Tex aren’t worth their contracts.

        Vote -1 Vote +1

      • Ludwig von Koopa says:

        I agree Yanks are still the favorite for #1… but I think Texas is closer than many give them credit for. (And I certainly don’t mean to imply that anyone is stronger financially than the Yankees, just that their advantage might not be insurmountable for the purposes of this ranking.)

        On the other hand, I agree that Houston is a lock for #30.

        Vote -1 Vote +1

  9. chuckb says:

    What about auctioning off the #6 spot for a donation to the leukemia research facility or organization that received donations for the Ottoneu league?

    Vote -1 Vote +1

  10. adohaj says:

    I like all of the changes quite a bit. Especially 2 and 3

    Vote -1 Vote +1

  11. Baltar says:

    Great changes, great joke.
    This should weed out some of the most obvious mistakes from last year, such as the too high placement of the Baltimore Orioles.

    Vote -1 Vote +1

  12. Jeff says:

    Does league status matter? I guess I mean with Houston moving out of the NL and into the AL, does that affect the future outlook? What might look good (in terms of talent) for the NL central might not so much now with Texas and LAA in the AL West.

    Vote -1 Vote +1

    • Jeff says:

      Not that what Houston has looks good, just that it might actually work worse by comparison.

      Vote -1 Vote +1

      • Ludwig von Koopa says:

        Well, compared to Oakland and Seattle, they don’t look horrible. Though they still look to be the worst of the three laggards.

        In related news… Could both Texas and LAA win 100 games this year? (Probably not, but it’s fun to speculate!)

        Vote -1 Vote +1

  13. jesse says:

    Have you thought about skipping number six and going frok 5 to 7 like hotels do with 13?

    Vote -1 Vote +1

  14. Sky Kalkman says:

    I’m collecting and compiling org rankings from us, the crowd. Go here to participate (directions included):

    http://goo.gl/WIeOc

    Vote -1 Vote +1

  15. Jack says:

    #6Org prediction: Arizona

    Vote -1 Vote +1

  16. Bronnt says:

    I’m glad to see Change #2. That’s the big problem I had with the series last year-pretty much half the league “tied” in terms of their future talent rankings, and the definition of “Future Talent” was extremely poorly done. Teams that have recently graduated a ton of prospects that are still under extended team control have excellent future talent, not poor. Change #3 also makes a lot of sense to me. Teams that are near the bottom in front office personnel might end up making some crippling decisions, but those teams also tend to have a high turnover rate in front office personnel, which changes the outlook.

    I’m looking forward to seeing how these changes improve the series. Reading your methodology, I don’t think that I am going to have any complaints. This is a good series, and I think now, in year 3, you’ve really ironed out the methodology.

    Also, thanks for not doing this earlier in the offseason, when I wasn’t reading Fangraphs every day.

    Vote -1 Vote +1

  17. Do new GMs get docked for a prior regime’s signings? I can see how you might say yes, if the owner (or powerful manager) is still there, but an owner (and manager) shouldn’t get negative marks for not meddling with or vetoing personnel decisions. Without visibility into each trade/signing, I don’t know how you answer that.

    I’m thinking of the Angels.

    Vote -1 Vote +1

  18. Eric B says:

    Great job on all the methodology changes! I thoroughly enjoyed reading last year’s rankings and expect these improvements will spur better debates instead of 500 questions as to why every other organization is ranked 3rd in future talent.

    Vote -1 Vote +1

  19. poor mets says:

    let’s just be honest here. the mets have to be the #6org because they are just in that bad of a situation.

    Vote -1 Vote +1

  20. cpebbles says:

    So when this exercise in undermining objectivity once again proves a hit to the credibility of this site, are you going to consider maybe giving this up next year?

    Vote -1 Vote +1

    • Jason B says:

      How exactly does this totally subjective, “meant as entertainment/thought exercise only” in any way detract from the “credibility of this site”?? Please explain.

      *Sits down indian-style, prepared to listen intently.*

      Vote -1 Vote +1

      • cpebbles says:

        I guess I just imagined that whole #6org episode that was a dominant criticism of this site for so many months that Cameron ended up posting a 4-part series trying to defend the methodology.

        And, no, it is not intended to be totally subjective. However, the problem with this methodology is that the only way to differentiate it from a simple power rankings (Which would be better determined by simply publishing an aggregate of projected standings) is to throw in inherently subjective rankings of front offices and subjective assessments of minor league talent, which keeps producing embarrassing results.

        The authors at some point need to realize that maybe they just don’t have sufficient ability to evaluate complex systems for which there are no metrics, and just give up this concept. A simple aggregate of 2012 projections with a paragraph or two at the bottom where the authors can praise or condemn a minor league system and front office would actually be in FanGraphs’ wheelhouse, would be informative, and would be objective enough to cut down on criticism drastically.

        Vote -1 Vote +1

  21. Kellin says:

    Would it be possible to post these in a sortable list so we could compare team rankings without having to read each article to see who’s tops in finances, then Baseball Ops, etc.? Thanks

    Vote -1 Vote +1

  22. awy says:

    some concerns.

    1. these rankings are most interesting in what they say about the future as well as ‘functional competency’ of the org. the previous weighting was superior in this regard. to learn about present strength, which is akin to “present inventory” in some ways, you read the various season projections.

    2. more on the inventory point then, present talent has to be adjusted by efficiency of that talent relative to resources taken. somehow after seeing detroit at 6 i’m not seeing this adjustment.

    3. sure, management talent is out there. owners are not all smart enough or in the right mindset to reform their organization though. to the extent that owners are important in shaping the future of the franchise, you could reverse the fluidity of management talent logic and say that the fact in this fluid talent pool there are still a bunch of teams run by backwards FOs speaks to deeper organizational problems at the top.

    Vote -1 Vote +1

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>