Improving WPS

“All happy families are alike; each unhappy family is unhappy in its own way.”  — L. Tolstoy

You can say something similar about baseball games. All boring games are alike; but exciting games are interesting in their own ways. Every boring game has one team building up a big early lead, which is never threatened. But there are many ways to have an exciting game: the pitcher’s duel, the slugfest, the late-inning comeback, extra innings, all in various combinations. And in between them are the bulk of games that are simply ordinary.

All of which makes ranking exciting games a tricky process, at least compared ranking to how boring they are. How does one compare Game 7 of the 1991 WS (1-0 in 10 innings) to Game 4 of the 1993 WS (15-14 in 9 innings) on the same scale? They’re great in different ways.

Back in 2005 I created a system to do just that, a rating system based simply on the runs scored in line score. I may have been the Christopher Columbus of that new world. And ranking the games allows you to rate post-season series-es.

The line-score system did work in the sense that it could tell the difference between a great game and a good one, and between a good one and an ordinary one. But while the line score gives you the basic outline of the game, it was blind to the details of what happens DURING each inning. Zero runs scored in the top of the 1st rates exactly the same; whether there were three pop-ups, or if three singles were followed by a triple play.

Eventually I realized that Baseball-Reference.com (ALL HAIL BBREF) has the play-by-play data for all playoff games, which includes a probability of victory after each play (anything that changes the outs, baserunners or score). Plotted, you can easily see if a game was good; It looks like and earthquake. If it was bad, it looks like the EKG of a corpse. Using those probabilities, we can create a much more accurate game rating. I fiddled with many rating schemes over the last 10 years before settling on one that seems both conceptually simple and that yields reasonable results.

Of course, by then I had been beaten to the basic concept by Dave Studeman (WPA) and Shane Tourtellotte (WPS). Twelve years is too long for laurels resting.

WPA = Sum(change in probability between plays)

Modified WPS = Sum(change in probability between plays) + top three plays + Final play

What I have developed is similar to their work, but I think it has some small advantages. Generally, my ratings will be quite close to Shane’s (R-squared > 99.5%). He correctly realized that simply summing the probabilities doesn’t quite work, which is why he modified it. An example…

There are seven post-season games with a WPA of exactly 4.52. Among them are:

1995 NLCS Game 2

Reds beat the Braves 6-2 in ten innings.

95 Plays, 13 plays changed the odds by at least 10%

top Play a Mark Portugal bases-loaded wild pitch +18%

70 plays with the odds in the 30% to 70% range

compared to

1960 WS Game 7

Pirates 10 Yankees 9 in nine innings

77 plays, 15 plays changed the odds by at least 10%

Of those 4 changed the odds by at least 20%

Of those 3 changed the odds by at least 30%

Of those 1 changed the odds by more than 50%

25 plays with the odds in the 30% to 70% range

 

There is simply no way those games are equal. The 1960 game has five different plays better than any play in the 1995 game. The 1995 game makes up the ground by (1) having 18 more plays (2) having fewer plays where nothing happened because the game was usually within one run.

1960 is still better because a +40% play isn’t twice as exciting as two +20% plays. Bill Mazeroski’s game-ending homer rates as +37%. Bobby Richardson’s game-starting line-out rates at +2%. Making a walk-off homer the equal of about 3 ½ innings with zero hits. NOPE. WRONG.

Shane accounted for this with his modified method. By counting the top three plays twice and Mazeroski’s walk-off homer three times, the ratings are now

1960: 6.49

1995: 5.19

And science prevails.

Of course, there is nothing magical about TOP THREE plays or LAST play. You could try using the top five plays and last five plays (believe me, I did).  But I do think that using Top-3 + Last can sometimes lead you astray. I will now present exhibits A and B to demonstrate where it can swing and miss.

Exhibit A: 1988 WS Game 1

Exhibit B: 1985 NLCS Game 6

I expect you to know them. The two biggest home runs in terms of changing the odds in post-season history courtesy of Mr. Clark and Mr. Gibson.

1985: WPA 4.48 in 83 plays and 9 innings

1988: WPA 3.94 in 82 plays and 9 innings

The 1985 game had more action with the same number of plays, which you can easily see in the line scores

StL          0              0              1              0              0              0              3              0              3              (7)

LA           1              1              0              0              2              0              0              1              0              (5)

 

Compared to

 

Oak        0              4              0              0              0              0              0              0              0              (4)

LA           2              0              0              0              0              1              0              0              2              (5)

 

The ‘85 game has a game tie in the 7th, broken tie in the 8th and lead change in the 9th

The ‘88 game has a lead change in the 2nd and a lead change in the 9th

Modified WPS says

1985: 4.48 + 1.34 + 0.01 = 5.83 (Tied for 94th best game)

1988: 3.94 + 1.43 + 0.87 = 6.28 (Tied for 58th best game)

I don’t think you can argue that the 1988 game is much better than the 1985 game; I don’t think it’s a better game at all. And it’s the last-play bonus that is to blame. Had the 1985 game been played in St. Louis then Clark’s homer would have been a walk-off and the game would have rated 6.56, well ahead of the 1988 game.

If you think about it, a last-play bonus is biased towards games won by the home team. If the home team loses, the last play will rarely amount to anything.

Only 23 times has it been at least 20%. When the home team wins, it is at least 20% 122 times.

Only 11 times has it been at least 30%. When the home team wins, it is at least 20% 96 times.

I also know this because I tried last play, last five plays, and last ten plays in trying to construct a rating system. I also tried top five plays, top ten plays, all plays over 10%, WPA – .03 per play (yielding the bizarre result of games with negative excitement).

Eventually I tried a simple power transformation on EVERY play. First, I tried summing the squares of the probabilities changes, like any good statistician would.

When I did that, the 1985 game Rated 10th and the 1988 game rated 5th. Which is the wrong order, and both games are just rated too high. Then I tried other powers…the Goldilocks approach, looking for the one that was just right.

 

Power             Rank               Rank

2.0          1985       10th         1988       5th best game

1.9          1985       12th         1988       8th Best game

1.8          1985       15th         1988       20th Best game

1.7          1985       23rd        1988       25th Best game

1.6          1985       32nd        1988       36th Best game

1.5          1985       38th        1988       51st Best game

1.4          1985       53rd        1988       76th Best game

1.3          1985       61st         1988       104th Best game

1.2          1985       79th        1988       133rd Best game

1.1          1985       100th      1988       158th Best game

1.0          1985       116th      1988      185th Best game

Everything above 1.7 was eliminated since it rated 1988 better than 1985

 

Here’s some shorthand I’m going to use:

Game 6 of the 1985 NLCS: STL 7, LA 5 in 9 innings — WPA 4.48 (9-4-2-1)

Game 1 of the 1988 WS: LA 5, SF 4 in 9 innings — WPA 3.98 (5-2-2-1)

The 1985 game had 9 plays rated>= 0.1, 4 plays rated>=0.2, 2 plays rated>=0.3 and 1 play rated >=0.5

The 1988 game had 5 plays rated>= 0.1, 2 plays rated>=0.2, 2 plays rated>=0.3 and 1 play rated >=0.5

For a sense of scale, the average game is WPA 2.67 (4.89-0.88-0.33-0.03)

(You can check the examples listed below on BBRef to get more detail on each game)

 

Checking 1.7, both exhibits rated higher than

Game 2 of the 2017 WS: HOU 7, LA 6 in 11 innings — WPA 5.30 (10-5-3-0)

Game 1 of the 2015 WS: KC 5, NYM 4 in 14 innings — WPA 6.36 (16-3-1-0)

1.7 weights the big plays too much

 

Checking 1.6, both test games rated higher than

Game 6 of the 1986 WS: NYM 6, BOS 5 in 10 innings — WPA 5.14 (16-3-3-0)

Game 6 of the 1986 NLCS: NYM 7, HOU 6 in 16 innings — WPA 5.80 (11-3-2-0)

1.6 weights the big plays too much

 

Checking 1.5,

the 1985 game rated higher than

Game 6 of the 1986 WS: NYM 6, BOS 5 in 10 innings — WPA 5.14 (16-3-3-0)

The 1988 game rated higher than

Game 4 of the 2001 WS: NYY 4, ARI 3 in 10 innings — WPA 4.58 (10-3-2-0)

1.5 weights the big plays too much, but it’s getting hard to find clear mistakes

 

Checking 1.4,

the 1985 game rated higher than

Game 3 of the 1976 NLCS: CIN 7, PHI 6 in 9 innings — WPA 4.72 (14-3-2-0)

Lead changes in the 7th, 8th and 9th innings.

The 1988 game rated higher than

Game 4 of the 1986 ALCS: CAL 4, BOS 3 in 11 innings — WPA 4.64 (7-4-2-0)

1.4 weights the big plays too much, but I’m now splitting hairs

 

Checking 1.3, I like this one. Let me check 1.2

 

Checking 1.2,

the 1985 game rated lower than

Game 2 of the 1996 ALDS: NYY 5, TEX 4 in 12 innings — WPA 5.02 (8-2-0-0)

Game 2 of the 1990 WS: CIN 5, OAK 4 in 10 innings — WPA 4.50 (10-2-0-0)

1.2 weights the big plays too little. Famous games are losing to games without any highlights.

 

So, I think 1.3 is the sweet spot.

My rating score is = Sum((change in probability between plays)^1.3) *2

The *2 at the end is purely cosmetic. It allows the very best game to score close to ten.

 

With base WPA, Gibson’s homer (.87) is worth about 25x a normal play (.035). With WPS it’s worth bout 75x a normal play. Raising all the plays to the 1.3 power means that Gibson’s homer is now worth about 65x a typical play.

With base WPA, Clark’s homer (.74) is worth about 21x a normal play (.035). With WPS it’s worth bout 42x a normal play. Raising all the plays to the 1.3 power means that Clark’s homer is now worth about 53x a typical play.

With a little algebra,

WPA:  Gibson = 1.18 * Clark

WPS: Gibson = 1.76 * Clark

Power 1.3: Gibson = 1.23 * Clark

A nice property of the transformation is that when the change in odds doubles, the play is worth ~ two and half times a much (2.46x)

 

EXCITEMENT IS NOT LINEAR

 

A 10% play is now worth 2.46 times as much as 5% play

A 20% play is now worth 2.46 times as much as 10% play

A 50% play is now worth 2.46 times as much as a 25% play

The system has a single parameter applied to ALL plays, so a game isn’t screwed if it has four great plays or the best play comes in the 8th inning. Ranking games this way, here are the five games better than, and worse than, my two test cases.

 

Series Road Team home team IP  (WPA^1.3)
*2
 WPA Top
Play
 # Plays  P>= .1  P>= .2  P>=.3  P>=.5
2014
ALCS G1
Royals 8 Orioles 6 10 5   5.14 35.0%         96        13         3         2        –
1935
WS G3
Tigers 6 Cubs 5 11 4.97   5.02 36.0%         96        15         5         1        –
1976
NLCS G3
Phillies 6 Reds 7 9 4.95   4.72 46.0%         82        14         3         2        –
2015
ALDS2 G2
Rangers 6 Blue Jays 4 14 4.93   5.46 37.0%       115         7         2         1        –
1997
ALCS G4
Orioles 7 Indians 8 9 4.92   4.92 38.0%         88        16         4         1        –
1985
NLCS G6
Cardinals 7 Dodgers 5 9 4.92   4.48 74.0%         83         9         4         2         1
1975
NLCS G3
Reds 5 Pirates 3 10 4.88   4.52 55.0%         81        14         3         3         1
1933
WS G4
Giants 2 Senators 1 11 4.87   4.94 55.0%         92         9         3         1         1
2011
ALCS G2
Tigers 3 Rangers 7 11 4.86   5.10 34.0%         92        13         3         1        –
2012
ALDS2 G2
Athletics 4 Tigers 5 9 4.86   4.86 41.0%         85        11         4         1        –
1999
NLCS G6
Mets 9 Braves 10 11 4.85   5.12 26.0%       108        14         3        –        –

 

 

Series Road home team IP  (WPA^1.3)
*2
 WPA Top
Play
 # Plays  P>= .1  P>= .2  P>=.3  P>=.5
1952
WS G5
Dodgers 6 Yankees 5 10 4.51   4.70 44.0%         92        10         4         1        –
1923
WS G1
Giants 5 Yankees 4 9 4.51   4.54 40.0%         78        12         2         2        –
1984
NLCS G4
Cubs 5 Padres 7 9 4.51   4.54 37.0%         83        10         4         2        –
1992
WS G2
Blue Jays 5 Braves 4 9 4.5   4.40 65.0%         85        11         1         1         1
1998
ALCS G2
Indians 4 Yankees 1 12 4.48   4.78 33.0%         96        11         3         1        –
1988
WS G1
Athletics 4 Dodgers 5 9 4.47   3.98 87.0%         82         5         2         2         1
2000
NLCS G2
Mets 6 Cardinals 5 9 4.46   4.66 32.0%         91        13         3         2        –
2016
NLDS2 G5
Dodgers 4 Nationals 3 9 4.46   4.66 21.0%         84        14         1        –        –
1977
WS G1
Dodgers 3 Yankees 4 12 4.45   4.80 30.0%         97        11         2         1        –
1954
WS G1
Indians 2 Giants 5 10 4.43   4.74 29.0%         89        11         1        –        –
1958
WS G1
Yankees 3 Braves 4 10 4.43   4.56 40.0%         88        10         3         2        –

 

 

I hope you’ll look at these and see that while they have different shapes, they all contain a similar ‘volume’ of excitement.

Another way to evaluate the method is to look at games with the same WPA. Going back to where I began in this article, here are the seven games with a base WPA of 4.52 (No promises that BBRef has not revised the scores since I captured the data…). They are each tied for the 108th highest WPA. But after using the 1.3 power factoring, you get this:

  Game Outcome RANK (WPA^1.3)*2  WPA   #
Plays
 Top 5
Plays
 # plays
30-70%
 P>=
.1
 P>=
.2
 P>=
.3
 P>=
.5
1960 WS G7 Pit 10 NYY 9 in 9 52              5.10   4.52     77   1.74         25    15      4      3      1
1975 NLCS G3 Cin 5 Pit 3 in 10 63              4.88   4.52     81   1.60         49    14      3      3      1
1911 WS G3 A’s 3 Giants 2 in 11 110              4.41   4.52     86   1.10         58    15      3      1     –
1998 NLCS G1 SD 3 Atl 2 in 10 117              4.36   4.52     84   1.10         59    11      2      1     –
2011 NLDS2 G5 Ari 3 Mil 2 in 10 119              4.35   4.52     85   1.05         71    13      2      1     –
1926 WS G5 NYY 3 StL 2 in 10 130              4.20   4.52     86   0.84         66    16      1     –     –
1995 NLCS G2 Atl 6 Cin 2 in 10 139              4.12   4.52     95   0.75         70    13     –     –     –

 

1960 gets the love it deserves, moving up 56 spots to the 52nd best game. That despite of having the fewest plays in the 30%-70% victory range. Games with more plays do worse since that means they have smaller impact plays on average. Think of the Top 5 plays as the highlight reel for the game. 1995 NLCS Game 2 has no play >0.2 and therefore drops 31 spots in the rankings.

Adjusted WPS? Weighted WPS? Power WPS? I really do need to give it a proper name.

 

A Final example, from among the greatest Playoff games ever.

2000 NLDS G3: Mets 3, Giants 2 in 13 innings — ModWPS Rank = 11, PowerWPS Rank = 22

1986 ALCS G5: Red Sox 7, Angels 6 in 11 innings — ModWPS Rank = 22, PowerWPS Rank = 12

1980 NLCS G5: Phillies 8, Astros 7 in 10 innings — ModWPS Rank = 25, PowerWPS Rank = 14

 

The 2000 game had the higher WPS, partly because it had more plays. ModWPS likes it more due to the additional action and walk-off homer, which the better top-three plays in 80/86 could not overcome.

 

year        WPS      Plays      Last Play    Top-3     ModWPS

2000       6.34        109         0.42          0.98                 7.74

1986       5.86       97           0.05             1.42                 7.33

1980       6.06        93           0.04            1.11                 7.21

 

So why do I think 1986/1980 are better?

Because, the deeper you go beyond the top three, the better the other two are revealed to be.

 

2000                                       1986                                       1980

1.28                                        1.94                                        1.61                        Sum of Top-5 Plays

42-31-25-16-14                  73-35-34-32-20                  40-38-35-26-24     Top-5 Plays

1.88                                        2.77                                        2.43                        Sum of Top-10 Plays

16-3-2-0                               14-5-4-1                               17-6-3-0               10%-20%-30%-50% plays

 

Or simply check the line scores.

2000

0 0 0 2 0 0 0 0 0 0 0 0 0 (2) Giants

0 0 0 0 0 1 0 1 0 0 0 0 1 (3) Mets

1986

0 2 0 0 0 0 0 0 4 0 1         (7) Red Sox

0 0 1 0 0 2 2 0 1 0 0         (6) Angels

1980

0 2 0 0 0 0 0 5 0 1             (8) Phillies

1 0 0 0 0 1 3 2 0 0             (7) Astros

 

The 2000 game IS a fabulous game. But the 1986 and 1980 games are more epic, with all the late-inning heroics. The 2000 game has exactly the required three big plays and the walk-off. It checks all the boxes.

I do kinda feel bad writing this. It sounds like I’m just picking on modified WPS here. LOOK AT WHAT ELSE IT GOT WRONG…

But as I said before, Power WPS is barely better. And to show that it’s better at all, I need to show those rare cases where it makes a better call. And it was an excellent benchmark, comparing differences between it and my sixty-eleven schemes helped me identify the flaws in sixty-ten of them.

Of course, even this is not the perfect system. Any play-by-play method will still fail to capture the in-play action. A bases-empty foul pop-out rates exactly the same as a bases-empty thrown-out-at-home-trying-to-stretch-a-triple. But it is the best we can do for now.

Whereas I used to guess my line score method captured maybe 70% of the excitement of a game, PBP ratings must be capturing upwards of 90%. Which means greater confidence in game rankings and playoff series ratings.

Anyway, if anyone has any thoughts, feedback, or questions I’d love to hear them. If no one can shoot the idea full of holes, or even one hole; then comes ranking and lists of games and series.



Print This Post

Comments are closed.