Leveraging the Orioles Bullpen
Last night the Orioles won another one-run game — no surprise there. Darren O’Day, Pedro Strop, and Jim Johnson combined for a scoreless 7th, 8th, and 9th — also no surprise. The Orioles bullpen is a major reason the team is tied atop the AL East. On Tuesday Jeff Zimmerman argued that successfully leveraging these relievers could partially account for the O’s performance in one run games. The theory is that an inconsistent rotation and streaky offense put the O’s in a disproportionate number of low leverage situations and preserving the best relievers hurts run differential in these games, while allowing the team to consistently use its best arms in close games. The result? Bigger blowout losses, fewer blowout wins, and more wins than typical run differential expectations would dictate.
Testing the theory that there is a method to the O’s 2012 run requires analysis of three questions. First, do the O’s find themselves in enough low leverage situations for such a strategy to impact their run differential on a significant scale? As Jeff pointed out on Tuesday, relative to other similarly successful teams the O’s seem to have a disproportionate number of blowout losses—games lost by more than six runs. In fact, the O’s have the 12th highest number of low leverage relief appearances in baseball.
Second, do the O’s actually employ a unique strategy in low leverage situations that suggests they are preserving the best relievers for higher leverage situations? Finally, does this strategy actually mean that the O’s best relievers appear in fewer low leverage situations and more high leverage situations than do the best relievers for other teams? The phrasing of this final question is very important, because every team tries to avoid using the top relievers in blowouts. The real question is do the O’s do so more successfully than other teams and, more importantly, is the result that these relievers consistently are available and successful in high leverage situations. The rest of the analysis will focus on these two important questions.
Surviving Blowouts: Low Leverage Extraordinaire Kevin Gregg
The first part of the theory that must be tested is whether or not the Orioles do in fact manage to avoid using the best relievers in low leverage situations. One method for doing so is to force starters to suck it up and push through bad starts to eat innings, but eventually this takes its toll and has long-term consequences. The table below shows all 11 of the Orioles losses by more than six runs (hereafter referred to as blowouts) this year in order of occurrence by the starting pitcher, the number of innings they threw, and the earned runs they allowed.
| Pitcher | Innings | Runs |
|---|---|---|
| Brian Matusz | 5.2 | 5 |
| Tommy Hunter | 5.2 | 8 |
| Brian Matusz | 5.0 | 7 |
| Jake Arrieta | 6.1 | 6 |
| Jason Hammel | 3.1 | 8 |
| Chris Tillman | 0.1 | 7 |
| Miguel Gonzalez | 2.2 | 7 |
| Zach Britton | 2.2 | 7 |
| Tommy Hunter | 3.0 | 8 |
| Joe Saunders | 5.1 | 6 |
| Zach Britton | 3.1 | 5 |
While there are some cases of pitchers gutting it out for a while, in all of these blowouts there were obviously a significant number of low leverage innings thrown by relievers. I don’t think we can say definitely that any blowout strategy employed involves hanging starters out to dry longer than other teams normally do.
So where do the Orioles’ low leverage relief innings go? To Kevin Gregg, of course. At .49, Kevin Gregg has the second lowest gmLI (average leverage index upon entering the game) of any reliever with at least 40 appearances in baseball. He has thrown in nine of the Orioles 11 blowout losses and two of their four blowout wins. In the process he has posted a very pedestrian 4.74 ERA, but has managed to grind through 43.2 innings in 40 appearances. He is not alone in the mop up ranks within elite bullpens, though. JP Howell for the Rays, Alfredo Simon for the Reds, Cristhian Martinez for the Braves, and Louis Coleman for the Royals have all made 40 or more relief appearances and have a gmLI less than .70 pitching for bullpens in the top ten in WAR.
Kevin Gregg is not the end of the story, however, and in fact it is the myriad of other pitchers the Orioles have used in low leverage situations that is most impressive. The team has a combined 54 relief appearances from 12 different pitchers including normal starter Tommy Hunter, acquisitions Randy Wolf, Miguel Socolovich, and J.C. Romero, and journeymen call ups like Dana Eveland. Of these 54 appearances, 37 (69%) of them have been in low leverage situations. All told the Orioles have used 19 relievers this year (if you include Chris Davis’s emergency extra inning appearance). Only ten teams have used at least 18 relievers this season and of these teams, only the Orioles and Red Sox rank in the top 15 in baseball in reliever WAR, implying that for the other teams it has a lot more to do with ineffectiveness than with strategy.
Leveraging O’Day, Strop, and Johnson
So the Orioles have been creative in constructing the relief corps, but how does that translate into actual usage of the team’s best relievers? First, a quick return to the 15 blowouts the Orioles have taken part in. Below is a table of the Orioles seven most used relievers (including Kevin Gregg) with their appearances in blowout losses, blowout wins, and all blowout games.
| Pitcher | Blowout Loss | Blowout Win | Total |
|---|---|---|---|
| Jim Johnson | 1 | - | 1 |
| Pedro Strop | 1 | - | 1 |
| Darren O’Day | 4 | - | 4 |
| Luis Ayala | 3 | 1 | 4 |
| Troy Patton | 5 | 1 | 5 |
| Matt Lindstrom | 5 | 1 | 6 |
| Kevin Gregg | 9 | 2 | 11 |
The Orioles thought enough of Matt Lindstrom to trade him to Arizona and a fair number of Troy Patton’s low leverage appearances came early in the season when he was somewhat less established. That leaves 10 total appearances in blowouts by Jim Johnson, Pedro Strop, Darren O’Day, and Luis Ayala in the eleven blowouts. However, that actually overstates the case, as half of those 10 appearances came during the stretch immediately after the All-Star break, where the Orioles played 20 games 20 days. During this stretch, the Orioles were at some points forced out of necessity to get innings out of these pitchers simply due to the lack of off days. Even including these games Jim Johnson and Pedro Strop have only pitched in one blowout each. Strop’s came after three straight rest days when he needed work. Johnson’s was a strange departure from the strategy on July 16th when he gave up 5 ER in 1/3 of an inning in a throwaway loss to Minnesota.
Now, the important point of all this is that the Orioles should be able to use their top pitchers in the most high leverage situations and in the least low leverage situations. I very subjectively chose the top three relievers from each team and looked at the percentage of each reliever’s appearances that were low leverage and the percentage that were high leverage. I also looked at the proportion of a team’s high and low leverage appearances accounted for by these top relievers.
The Orioles seem to do pretty well within this framework, but there are a handful of other teams that leverage their best relievers similarly. The average proportion of total appearances that were in low leverage situations among O’Day, Strop, and Johnson was the 9th lowest in the league. These three relievers made up 28% of the team’s total relief appearances in low leverage situations, good for the 11th fewest in the league. More importantly, the average proportion of total appearances that were in high leverage situations for the Orioles relievers was second highest only to the Royals (for whom I included Jonathan Broxton, despite being traded, because without including the team’s closer for much of the season it skews the results). These three relievers made up 63% of the Orioles total high leverage appearances, good for 6th in the league.
As far as minimizing appearances in low leverage situations and maximizing appearances in high leverage situations for the top three relievers, the Giants, Indians, Reds, and Royals are the only teams that seem to do as well or better than the Orioles across the board. As they did last night, the Rays rely heavily on Joel Peralta, Jake McGee, and Fernando Rodney in high leverage situations, but also use them in a fair amount of low leverage situations.
If we restrict our analysis to only the team’s top two relievers the combination of Jim Johnson and Pedro Strop rank 3rd and 30th in the league in gmLI, respectively. Here only the Rockies, Reds, Indians, and Giants have a higher combined ranking among their top two relievers. In fact, the three headed monster of Javier Lopez, Santiago Casilla, and Sergio Romo in San Francisco has a combined higher ranking with gmLI’s of 6, 10, and 11, respectively.
Strategy or Luck?
In a way this is circular logic, but baseball’s best bullpens all do a good job of leveraging their best relievers and the Orioles seem to fit squarely into this group. What appears to be different is the manner in which the Orioles have creatively found ways to avoid using these relievers in blowout games, which in turn has caused the team’s run differential to take a hit.
I do not think this is the whole story — certainly a combination of randomness and offensive performance has a significant impact on winning close games as well. I do think, however, that Dan Duquette and Buck Showalter deserve credit for being creative and sticking to a potentially unpopular method for utilizing an already great bullpen. Having been a part of teams that felt like the coaches quit on games that got out of hand, I cannot emphasize enough the importance of Showalter’s ability to communicate this strategy with players to prevent it from becoming a negative.
As Jeff Zimmerman pointed out on Tuesday, whatever the true reason for the Orioles unlikely success, barring a major collapse in the final weeks we could be witnessing the greatest “over-performance” in baseball history — or perhaps one of the greatest strokes of managerial genius.
I was curious as to whether Buck Showalter’s teams have a tendency to outperform their WAR, given his (absurdly consistent) history of turning losing teams into playoff teams in his second season.
Overall, Buck’s teams have actually underperformed their WAR by 5.8 wins. Disclaimer: I was too lazy to figure out actual replacement level win totals, so I used 43 wins and prorated the strike season. Also, I ignored the 50 some odd games he coached in 2010, because, honestly, does anyone care about the 2010 Orioles? Figuring out accurate replacement team numbers would almost certainly result in worse numbers for Buck.
Before 2012, that number turns into -16.8 wins over 12 years. So I guess if we decide that the O’s outperforming their true talent is Buck’s doing, then we probably have to come to terms with the fact that Buck has been a bad manager when we survey his entire career.
Curiously, of the five seasons out of 13 that Buck’s teams have outperformed their WAR, three have been his second year with a team.
When you look at his Pythag versus his actual wins, he comes out at 5 wins above expected (-6 before this season).
My conclusion? Managers are overrated. Also, there are too many baseball stats, which makes my head hurt.
Very interesting article, thanks. Where/when did you play? Just curious.
I just finished playing 4 years at Brown. As a disclaimer, the coach comment refers to various teams at various levels…just in case anyone wanted to try to read into it further than that.
The O’s have been a big puzzle for me, I haven’t been able to figure this team out at all. If you look at their offense and their starting pitching, it’s pretty mediocre. OK the pitching is better than the offense.
To me there is just no way that this team should have this many wins.
The Orioles are 64-0 when leading after 7 innings–so they condense the game. They are 3rd in MLB in home runs–so they can change the complexion of a game on one swing. In a condensed game, any runs (especially home runs) are like gold.
OK terrific my question is why are they 64-0 and those situations
* in * those situations (Typo)
In other words just clarify why does a basically mediocre team put itself in the position of leading after 7 innings so often
I’m not sure having a lead after the seventh in 64 out of 143 games is that good. That means in 79 out of 143 games, they’ve been either tied or behind after the seventh.
My guess is that it’s the starting pitching. Either the starters get rocked or they put together a good outing (but not necessarily going six innings). The offense is designed to score less than the average but not get shut out (lots of power and bad OBP). If you can put together a good pitching outing and consistantly get a few runs each game, you’ll win a lot of low scoring games.
I could be wrong though. I’m not sure how to test it with the data that I have.
I think you missed the most important question:
Is there enough of a difference between the Orioles “bad” relievers and the Orioles “good” relievers for it to matter which ones pitch in low leverage situations?
Given that Patton, Lindstrom, Strop, JJ, O’Day and Ayala all have ERAs under 3 it means that it doesn’t matter which ones are pitching in high-leverage or low-leverage situations because they’re all effective. The only way leveraging matters is if Gregg, Eveland and the Os bad relievers are so bad that they have a substantial impact on bullpen performance.
To be fair, it seems that they do. The Orioles high leverage relievers have an average ERA of 2.5 while the low leverage guys have an average ERA of 4.23.
But there’s only one small problem. The Rays also use their relievers well. Their high leverage guys have an ERA of 2.16 while their low guys have an ERA of 3.32. Given your argument, you’d expect the Rays to be awesome in one run games. Unfortunately, they’re 20-26.
The difficulty of arguing that the Orioles bullpen is helping them win one run games is the fact that the Rays bullpen (with a better performance) isn’t.
Really great comments and I appreciate the stats. A few follow up points:
You’re right that a really important question is whether the O’s mop up guys really are significantly worse—and in fact they are. It was something I looked at, but didn’t put into the article in detail (as I should have) so thanks for the actual stats.
That point kind of segues into a larger overall point about the argument as a whole. Bullpen usage is obviously not the entire story, especially as far as winning in close games is concerned—the fact that the O’s are second in baseball in wOBA in high leverage situations (Rays are 20th) clearly matters a lot. Also, as I mention in the article, they don’t leverage the top guys that differently than do most of the other top bullpens—though the Rays run the top guys out a little more often overall, including in more low leverage spots. I think what’s unique about the O’s is that they have been creative in filling those low leverage situations (in some cases with pretty subpar arms) because they really do not have good, consistent options for that role. Given their rotation they likely have more of those situations as well, but they haven’t let that compromise their ability to use the high leverage guys when they need them.
Looking at the Rays specifically, if we consider J.P. Howell the team’s equivalent of Kevin Gregg, he’s got an ERA basically two runs lower than Gregg (3.00 vs. 4.95). Also, as you mention, the Rays bullpen is better overall, the mop up guys as a whole are better, and the difference between the mop up guys and high leverage guys in ERA is smaller. As far as winning close games, the difference in the O’s and Rays is probably offense and “luck”. The reason Tampa is underperforming its run differential and the O’s are vastly outperforming it, though, may in part (small part?) be these low leverage options. In other words, if you give the O’s the Rays’ offense and luck, of course they win fewer close games and have a true record closer to Pyth. record; but they still could outperform run differential because their starters put them in more low leverage situations and the O’s don’t have the quality of low leverage options the Rays do—which again highlights the importance of your first point.
The reason I started looking at this is I think there’s an interesting relationship between how a team leverages relievers, success in close games, and overall run differential. Outperforming run differential obviously requires both that a team give up a lot of runs (relative to how many they score) and that they win a disproportionate number of close games. Leveraging relievers can affect both ends of that equation. I think for the O’s the combination of an inconsistent staff providing lots of low leverage situations, few good options in low leverage situations (I don’t consider Kevin Gregg’s 4.95 ERA a quality option, but it’s better than using Darren O’Day to mop up), and great back of the bullpen options equates to a perfect storm of these factors aligning for outperforming run differential. That being said, this is clearly not the only reason for their performance; frankly, if we are comparing the O’s to other teams (the Rays in particular), leveraging relievers probably has affected the runs allowed part of the equation more than it has affected their disproportionate success in close games.
What the Orioles are doing is historically unprecedented. I dont think people fully grasp that.
Using baseball-reference.com, I looked at every single MLB team from every season from 1901-2011. that’s 2,302 team-seasons. Exactly 14 teams finished with a 1-run game winning percentage of .700+ The all-time record is held by the 1981 Orioles at .750. if you think that a strike-shortened season shouldnt count, your team is the 1908 Pirates, at .733. [data here] And how are the 2012 Orioles doing? They currently have a 27-7 record, for a winning percentage of .794
The 2012 Orioles are about to set the all-time record for 1-run game winning percentage – and by a lot.
That tells me that everything is going right, including a ton of luck. The fact that on the season the Orioles have scored fewer runs than they have allowed makes it even more jaw-droppingly absurd.
Yes exactly. On paper they shouldn’t be winning this many games but they somehow manage to scrape by, and actually take series from the Rays, Yankees, etc., who we would all agree have much better teams on paper.