Translating Farm System Rankings into Wins
Knowledgeable baseball fans clamor for rankings of farm systems every offseason. Experts’ opinions are highly coveted, as fans eagerly await information on where their team’s prospects rank. This year, Baseball America ranked the Rangers atop its list, with the Royals just behind them. On the other hand, The White Sox’ and Indians’ farm systems were ranked at the bottom.
This year’s rankings saw big spenders like the Phillies, Dodgers and Tigers in the bottom-third in baseball, while the top-third included some teams with the lightest payrolls — like Rays, A’s and Padres. Obviously fans want their teams to be big spenders on veterans — and big farmers of young talent — but which is more important? What does a good ranking mean in the win and loss columns, and how much can payroll explain that performance?
Team building is at the heart of baseball analysis. And analysts like those here at FanGraphs try to deduce what teams should do to get themselves into the playoffs. Recently, I wrote about a couple of important team-building concepts at The Hardball Times. Specifically, I separated all WAR into either “Non-Market WAR” and “Auction-Market WAR.”
Non-Market WAR (NM WAR) refers to all players who weren’t yet eligible for free agency — so they were either making league-minimum salaries or arbitration salaries. Auction-Market WAR (AM WAR) refers primarily to players with at least six years of service time. The only exception is professional free agents, typically from countries in Asia or Cuba, since teams aggressively bid on them like free agents. The difference is that teams had to consider other teams as competition when paying for AM WAR — but NM WAR is homegrown.
Each year, free-agent-eligible players get about 75% of payroll, but they only produce about 30% of all WAR. The average team gets about 12 WAR from Auction-Market talent, but 26 WAR from Non-Market Talent. At the current price of free-agent talent, it’s effectively impossible to build a team out of auction-market talent alone. We already know that a team of replacement-level players would only win 43 games. The 2009 Yankees would be the only team in the past five years to have enough AM WAR (41.0) to be above .500 without any contribution from NM WAR—but they would fall short of the playoffs at 84-78. Put simply: You absolutely need some cheap talent to win.
On the other hand, there have been 13 teams in the past five years with more than 38 NM WAR — which is enough to break .500 without dabbling in free agency. The best of these teams was the 2011 Rangers, who got 47.8 WAR from players not yet eligible for free agency. Adding this to the 43 wins that a replacement level team would put together, that means that the 2011 Rangers could have reached 91 wins without spending a dime on free agents. This is a strong contrast with the 2007 White Sox, which only had 5.3 NM WAR. That means that even with the 2009 Yankees’ free-agent production, the 2007 White Sox still wouldn’t have won as many games as the 2011 Rangers’ amateurs would have won if you replaced all players with six years of service time with AAAA talent. So you can certainly make good use of free agents’ AM WAR, but it’s possible to win big with NM WAR alone. You just need to be both very lucky and very good at scouting and development.
Building from within is essential to team success, and that’s why farm-system rankings are so important to knowledgeable baseball fans. Do they tell us that the Rangers are the premier team to compete with over the next few years? Are the Royals the next juggernaut in the AL Central? To answer this, the next step is to quantify these rankings. To figure this out, I gathered the farm-system rankings from Baseball America for the past 10 years.
Sure enough, there is a sizable and obvious correlation between Baseball America farm system rankings and NM WAR in subsequent years. Baseball America ranked the Rangers fourth, first and second between 2008 and 2010, and the team subsequently led the league in NM WAR in 2011. The biggest producers who factored into the 2011 rankings were Elvis Andrus and Matt Harrison, but their NM WAR leaders were Ian Kinsler, Mike Napoli and C.J. Wilson.
The problem is that much of the variation in NM WAR is still unexplained—at best, about 7% of differences between teams’ NM WAR in a given year can be explained by differences in prospect rankings. The rest is either something that prospect gurus were missing or something that could not have been foreseen. Either way, rankings can explain a limited amount of this very important difference between teams. The Diamondbacks, for example, had more WAR from players who did not reach arbitration eligibility than any other team in 2011 (23.1 WAR). At the same time, their farm systems were ranked third, 15th, 26th, 27th and 22nd between 2007 and 2011. The Twins were ranked a respectable eighth, 18th, 22nd, 7th and 13th in the past five years, but they only had 11.5 NM WAR in 2011 —which was lower than any other team last year.
If you regress NM WAR in each of the next five years against a team’s Baseball America farm system ranking, you’ll find that the difference in NM WAR between the best and worst teams in the league is about 41 expected wins over five years. In other words, the difference between the Rangers and White Sox between 2012 and 2016 should be about 41 wins — plus or minus differences that can be expected from major league contributors and Auction-Market WAR. The biggest effect is two years later, where the gap is 10 wins (31 vs. 21 NM WAR). Of course, that’s the expected NM WAR based on rankings. Actual differences are much larger with all factors considered — the difference between the Rays and the Mariners, the best and worst teams by NM WAR in the past five years, was 122.1 WAR (194.2 and 72.1). It’s just that farm-system rankings weren’t going to tell you what the Rays would do that the Mariners would not.
So we have some numbers: Each team can expect about 1.4 more wins in the next five years than the team ranked below it, with the largest gap occurring two years later with about 0.33 wins difference for each spot in the rankings. That’s definitely important, but it’s hardly a crystal ball.
Contrary to past farm system rankings’ limited ability to predict future NM WAR, past payroll predicts future AM WAR very well. If you take payroll from 2007 and try to predict AM WAR for 2007 through 2011 through regression, the differences between the highest and lowest (Yankees and Rays) expected AM WAR goes down from 41 WAR in 2007 to 25 WAR in 2011. Overall, the difference is about 160 WAR in those five years. Even excluding the Yankees, the difference between the Rays and the second-highest team (Red Sox) is about 115 AM WAR.
Adjusting for payroll growth over the last five years, every extra $1 million of payroll today suggests about 0.74 more AM WAR in the next five seasons, combined. Annually, this effect starts at 0.19 more AM WAR this year per $1 million, down to 0.12 more AM WAR in four years.
Bringing these two things together, we see that differences between the top and bottom teams in farm ranking will accumulate to about 41 WAR over 5 years, with the biggest different being about 10 wins two years after the ranking. The difference between the top and bottom payroll rankings will be about 160 WAR over five years, gradually decreasing from 40 WAR that same year to 25 WAR four years later.
So if I were to tell you that you had the fifth-highest payroll in 2011 and the fifth-worst farm system, chances are you would get five more AM WAR in two years than the average team and three fewer NM WAR in two years than the average team in two years —which would put your team around 83-79. If you were first in farm and last in payroll, you might expect to finish with five more NM WAR than average and about 13 fewer AM WAR than average — which would relegate you to a 73-89 finish.
That means it’s better to be the Phillies right now (third in payroll and 27th in farm, implying an expected difference of +11 AM WAR, -4 NM WAR, versus average) than the Royals (25th in payroll and second in farm, suggesting -6 AM WAR, +4 NM WAR, versus average). Of course, you’d rather be the Red Sox than either (second in payroll and ninth in farm, +11 AM WAR, +2 NM WAR, versus average), and you definitely wouldn’t want to be the Indians (24th in payroll and 29th in farm, -7 AM WAR, -4 NM WAR, versus average).
Payroll in 2007 can tell you about four times as much as about the difference between teams over the next five years as farm-system rankings can. As bright as the Rangers’ future may look, the Yankees future is probably a bit brighter. Even still, 41 wins difference between the top and bottom teams is not small. If the White Sox were to ask the Rangers to swap farm systems, a fair price would be about $235 million. That’s useful, but the foundation of a big payroll — a large coastal city full of deep pocketed fans, a regional sports network and a retro-classic stadium— is worth a lot more than that. And now we have some numbers that tell us about how much.
This is a really, really good post, could be titled “How to be a GM 101″, and then forwarded to Ned Coletti, Jim Bowden, Kenny Williams, etc.
aren’t two of those teams in first place?
having said that…they could certainly benefit from this terrific read.
“This is a really, really good post, could be titled “How to be a GM 101?, and then forwarded to Ned Coletti, Jim Bowden, Kenny Williams, etc.”
Maybe Kenny Williams can read it after he gets done polishing his World Series championship ring.
Haha, you think the Sox won the WS because of moves Kenny Williams made. That’s…well, you made my Monday a little better.
Or Jerry Reinsdorf.
Great post-sadly Reds fans are feeling the pressure of having a farm system which at this point is in the bottom say 25th in baseball–Krivsky had them in the top 33 percent–thanks be to Jocketty and an owner who put him in charge Castleinni Red fan since 1950 sorry they still are my team.
The Reds are not in the bottom 25th. Their system is still strong, just not elite. With Mesoraco graduating, Billy Hamilton becomes the top prospect. If he continues near his current levels, he will fly up the rankings like he flys around the bases. Didi Gregorius is more highly thought of than Cozart. Donald Lutz has big time power at a corner OF spot. And Henry Rodriguez and David Vidal will fight for third base in the future. They are also have power arms Corcino, Cingrani, Lotzkar, and Sulbaran, plus first round pick Robert Stephenson.
Check out so-called prospects with what has been to this point 2012 yeh Mawwell chance
How does the ability to give extensions affect the categories? (especially high dollar extensions to players such as Tulo, Mauer, etc.) These players aren’t at minimum or arbitration salaries but also haven’t been exposed to the auction market of free agency.
Would a mixed market or indirect market category improve the predictive abilities of this system?
I’d bet they balance out. After all, a typical extension is given during arb-years, even if it extends into FA years. So usually what you’d get, I think, would be an overpay for NM and an underpay for AM, which should balance out.
Ability to give extensions probably is part of payroll’s predictability of AM WAR. For extensions that buy out arbitration years, that’s usually just a team acquiring cost certainty and a player getting security.
Really great. After reading the headline, I expected the body of the piece to be ‘is really hard.’
Excellent article. As a White Sox fan, I’m just going to sit in the corner and be depressed.
A couple of issues/questions:
Should one try to account for the NM WAR of players from within a system, rather than players who wind up playing for the team? If a team trades a prospect for a vet, then the prospect will accumulate his NM WAR somewhere else, but it’s the strength of the farm system that allowed the trade for the vet.
Also, should you scale NM WAR to NM playing time? Some teams will get less NM WAR because they’re giving PT to free agents. On the other hand, some teams have to give PT to free agents because they don’t have good enough players in the farm system, so it’s hard to measure.
I addressed the issue of NM WAR that gets traded away a little bit in the THT article linked within, but it’s mostly a smaller issue in this piece so I ignored it.
Playing time is mostly a function of ability. It’s a good point, but I think it’s minor in terms of making general claims about estimates. It does play into the trades issue though you addressed in the first paragraph.
Very nice piece, we all know that more money lends an advantage, but this is a solid factual look at it. So, payroll is 4 time more important than farm system ranking. Its beyond me why the smaller to mid market owners don’t revolt against the system. Instead they quietly nod along and let Bud eliminate the few loopholes out there to build a better farm system.
The lesser organizations refuse to revolt against this economy because most make good money from this system. Briefly put, MLB money transfers to low-revenue teams amount to bribes. The low-revenue teams receive monies meant to induce them to refuse to break the money machine which is MLB. They stay quiet and make do with their limited opportunities.
The deadweight losers in this ‘game’? The fans of the low-revenue teams. They pay too much to receive less baseball value than they would receive if the baseball game were fair. But the baseball game is not fair. It does not measure FO competence. It strongly reflects the economic resources of the various baseball franchises, resources which issue from team locations, television revenues, etc.
Your perspective is interesting and you make a very good point– revenue sharing is a way of keeping teams from revolting against the system. It’s a transfer that can create a close to optimal result from the perspective of the teams alone.
In terms of the fans, it’s tricky. The transfer of utility clearly goes from small market fans to large market fans, but that doesn’t necessarily mean it’s a deadweight loss. If there are more fans in NYC, then a higher percentage of fans see their team win a championship with the current system than a system where Kansas City wins as often as the Yankees. So you could make an argument that the social good is for the Yankees to win a lot but not so often that it’s not interesting for all involved. Kind of a murky area, and comparing utility is a tricky venture.
Of course that doesn’t address situations like the Marlins a couple of years ago, who were spending less on payroll than they were receiving in transfer payments, with the difference going straight into Jeffery Loria’s pocket. There’s not much utility to that for anyone, except Mr Loria. I’ve always been surprised that the MLBPA doesn’t make a bigger issue out of teams spending every dollar of the transfer payments, luxury tax, and shared merchandise revenue on player salaries, which would obviously benefit their members but would also (eventually, hopefully) pay off for the fans of those teams.
One reason your regression explains so little is that system ranking collapses too much information into a single number. Is the system stronger in the upper or in the lower levels? A better predictor of NMWAR in a single year is likely to be the number of good prospects in AA and AAA in the preceding two seasons and the number of good prospects in A-level or lower three to five years earlier. Statistically, you want to do a time-series analysis with lags for numbers of prospects in each level by year.
You make a good point, but it kind of depends what your goal is. My goal was to provide an unbiased estimate of the effect of farm systems over several years, not a dead on model for the effect of each year specifically. So what I did was regress NM WAR same year, NM WAR next year, etc. until the farm system rankings stopped being predictive (about 5 years). And then I summed the coefficients to get the 1.4 WAR over five years. So even the single-A players in produce four years later, they are still included, and if the AAA players produce 0-1 years later, they are still includd.
OK, but you argue against the importance of farm system quality because it doesn’t predict NMWAR very well – there’s a lot of variance you don’t explain that could be explained.
True, that’s not the best NM WAR prediction that could be done if that was the goal. The cumulative WAR over the five year period that I talked about later was really a better description of that. Thanks.
I don’t comment very often and don’t have much to add. Just wanted to say that I thoroughly enjoyed both the article and the discussion in the comments that followed. It’s refreshing to see an overall team-building focused article. Additionally I would like to mention that I like the fact that baseball isn’t set up to be fair. The other pro north american sports are more focused on parity with salary (although in the end large markets still have an advantage.) I enjoy that baseball is a microcosm of society in the sense that all teams are not created equal and a whole litany of variables (market size, wealth of owner, competence of FO, luck, etc.) determine winning and losing. And this is coming from a fan of the team with one of the lowest average payrolls over its history.
Very true about society. You have the ones near the bottom who complain(ed) about being poor but basically leached off other people (this is the Marlins prior to this year). You have the smart, driven people who take the help they’re given and become awesome (Rays), you have the rich guys who still work hard (Yankees), and the rich guys who suceed pretty much soley based on their wealth (Phillies at this point) and then the people who suck at life and even though they have pretty decent resources aren’t successful at all (Chicago teams, probably not now that Epstein is at the helm).
That said, I’d like to see more parity. If the final standings are like they are today (no Phillies, Yankees, or Sox in the playoffs) I’d be a happy man.
Yep. That right there is DEAD on social analysis. Brwvo. I mean obviously our society is broken into about four categories with poor leechers and rich hard workers. You will find it alot more funto work hard when that work leads to a yacht. You are a fool sir and are almost certainly sokeone born on third base who thinks he hit a triple. Im sure you are getting set to regale me with stories of your grandparents (usually this actually is grea greatgeandparents) stirring immigrant success story. How theyworked 16hours tobuild wealth so their descendents wouldnt have to. Spare me.
Life is likebaseball but not how YOU think. Money in life is jyst like baseball. If you are rich enough it doesnt matterhl how fat lazy or stupid you are. No matter what mistake you make, whichevercrappy contract you give or college you flunk out of, you have another chance and another and another. If you are poor you get one shot and you have to be perfect because there is ZERO margin for error. A lost job leaves you 6 weeks from eviction. Aserious illness is fatal. A 16 year old arrest for pot means you cant get financial aid etc.
Talk to a poor person, and not just about how high you want your damn lawn mowed, before making idiotic social commentary. Hey, lucky for you I AM poor, so you can start here idiot. Just dont tell me im poor because i dont work hard.
Love this work. This higher-level team building perspective is too often reduced to rotisserrie style team-building which really has little to do with putting together an MLB roster.
“If you were first in farm and last in payroll, you might expect to finish with five more NM WAR than average and about 13 fewer AM WAR than average — which would relegate you to a 73-89 finish.”
You mean: If you are the Padres, you might expect…
Great read, thanks!
Nice work, this is very interesting, in particular the difference in how much teams get from NM payroll vs. AM payroll was really enlightening. I have two comments/questions.
1) To what extent are minor league system rankings biased by the perception of depth. It seems entirely reasonable that top talent in a system might predict major league success in a few years considerably better than system depth. Having Bryce Harper in your system with a bunch of D+ prospects might be considerably better for your team in the future than having a bunch of B- prospects and no stars.
2) Longstanding complaint with regressions on fangraphs. Can you please include the regression output? It adds clarity to how you set up the study, demonstrates that your predictor variables are statistically significant, shows us that the model is significant, and gives us a sense of it’s general fit. I don’t really know the answer to any of those questions from what you’ve written here other than a general sense that past payroll fits a little better than past farm systems.
I think what this analysis really shows is the value of high draft picks.
Good teams, especially teams that have been good for a long time (or perhaps more importantly haven’t been bad in a long time), are at a decided advantage in acquiring non-market talent, and are forced to either resort to purchasing market talent or trading non-market for market talent.
If we assume that the value of draft picks adheres to a Pareto distribution (as it should), the first pick in the draft would be twice as valuable as the second and twenty times as valuable as the twentieth.
Again, I think what your analysis really shows is that the value of draft position is severely under-rated. I also think the fact some of the small market teams have finally figured that out is why a number of big market teams are suddenly finding they have competition.
Oops, meant disadvantage (sorry about the typo!)
This article might be right up your alley. This is the basis for how I value individual draft picks in calculating $/WAR adjustments for compensation picks:
http://baseballanalysts.com/archives/2009/06/the_draft_and_w.php
Well here’s the issue with that. Teams at the top are weighting their payroll, and sometimes don’t want to give an overslot deal to a player at pick 8. This should be nullified by the CBA, but in the past:
Players like Josh Bell fell down into the 2nd round, Daniel Norris in the 2nd, Dillon Howard, etc. all were probable 1st rounders who were 2nd round picks. Of course they went to PIT, TOR, and CLE respectively, but they could have been taken by big name teams, paid an overslot deal, and gotten top 15 talent (Josh Bell).
Guys even later, like one of my personal favorites Clay Holmes (PIT rd 9), got a 2nd round deal to sign. So it’s not really a linear curve because of that added factor that players have extra weight whether they’ll sign or not.
If we were comparing a draft that way, it’d have to be a draft where the best players were taken from top to bottom, in order or close to it. With some of the most talented players going in the 2nd round or later, it’s difficult to call the line linear where Josh Bell is in the 2nd with top 10 talent.
This is awesome. The next logical step: take they payroll and farm ranking of every team and make a 5 year projection of wins. I’d love to see how that method stacks up against typical projection systems, which rely on the specific personnel and thus can only be done for the upcoming season.
This was probably the best fg piece i have read in months. Fantastic. This is the kind of serious analysis you really cant get anywhere else.
There is more work to be done here andi hope you perhaps dig even deeper in future posts. Hypothetically should we not in retrospect explain everything about a teams performance? Loved it.
Just read the THT piece. Highly recomended.
One more interesting thing in you THT tables is the team dollars per war from FAs. It is interesting to note that the real small market teams seem to get abysmal FA value while the big spenders end in the middle of the pack and the mid markets near the top (every FO generalization we make must always exclude the rays at this point). Basically this kinda puts the kibosh on the idea that the yankees and sox and phillies always overpay. What it seems to say is the big markets end up paying market value because they have the money and can also offer other things (great cities, endorsement opportunities, consistent contender status) while the smallest markets MUST overpay because money is their only currency. Many are in crappy stadiums, they have poor attendance, and have little hope of competing. While the mid markets like minnesota, dallas, st. Louis have the best of it sll. Theycan pay fair market, they can teasonably compete, they have good solid fans.
Interesting.
Awesome article, I will need to refer back to this a lot this coming offseason. Is the sample size big enough to account for divisional strengths / strength of schedule, so to speak? Part of the success of the Rangers’ farm might not be just the quality of their prospects in the abstract, but the soft landing of playing against the A’s and M’s so much in their initial ML seasons…
As a Cardinals fan, I find the situation very interesting.
I read an article elsewhere that showed StL gets the most WAR/$ in FA.
They also get good WAR out of “non-prospects” (Jay, Freese, Craig, Garcia), and when they do have highly rated prospects (which seems to be rare) they seem to trade them away, Rasmus for pitching, Drew for Wainwright, etc.
The biggest obstacle I see for prospect-based teams is that they need them to peak in the same window of time to cause significant change, versus teams that can just plug a good prospect into an already good team (BOS, ATL, etc).
KC is deemed to have possibly the best collection of prospects in recent history, and we’re seeing the problems they can present. If they could plug 1-2 batting and 1-2 pitching prospects into a solid team, they’d have something special. But when a team is dependent on needing 6-8 prospects to “really pan out”, they are likely to endure setbacks that ruin the plan. OAK’s “four aces” and NYM’s Izzy, Miller, and Pulsipher are other notable examples.
Teams like OAK, show just how difficult it can be to draft very well on a consistent basis.
It seems to be difficult to determine conclusion art statements on what wins out in the brains v. Brawn scenarios. Obviously it’s better to have both, but that’s not a realistic expectation for very many teams.
Yeah circle. Problem is you are waiting for all these ptospects to hit at once. But no matter what you MUST get market production and players do not sign with teams that MIGHT be contenders (not the good ones at least) so to be a viable destination for star players i think a small market team must overpay to supplement the non market core enough to convince star players that they could br the difference maker.
The cards are in a different spot because that city and fanbase are recognized as consistent contenders.
In this light the werth contract for instance is a totslly defensible move to build a reputation as a place to sign.
Even if they had the resources the royals would have pay so far over market to lure an impact star player.
Matt-
Great article. Quick question- you say that much of the variation in NM WAR is unexplained, only about 7% of difference IS explained. How did you come to this figure? What was the math behind figuring out how much of the variation was explained by differences in prospect rankings?
The 7% is actually the R^2 for this regression equation. The way it is calculated is basically that if you look at the best-fit equation of NM WAR = a + b*Ranking, and then create estimates of NM WAR for each team based on their ranking, and then calculated the variance of estimated NM WAR, it will only be 7% of the variance of actual NM WAR.