The problem with using simulations is that the programmer’s assumptions about how baseball works will be “validated” by the program. A baseball sim will have to either include or not include a protection effect, for example. Using the sim to then test lineup orders to see whether or not protection exists will just reveal the rules of the sim, not of baseball.
You can’t use sims to test fundamental laws of baseball without being INCREDIBLY careful.
Another thing to consider in making more efficient lineups is someone like Carlos Ruiz hit .302 last year and had a .400 OBP. But he did that hitting in front of the pitcher last year. If you’d put his “.400 OBP” leadoff, he would cease to have that high walk rate.
Great point Ryan. The Reds toyed with moving Ryan Hanigan’s .436 OBP up in the order last year, but in the games where they did his OBP went down by over 100 pts. Of course its a SSS, but it is only one example of what should be fairly obvious. Guys who use being pitched around to inflate their OBP should not be moved to a part of the order where they won’t be pitched around.
Simulations are only a good indicator of what to do with a fixed level of production to maximize runs. That isn’t what you’re doing with a lineup. You’re trying to maximize production first and foremost. Utilizing that production is somewhat important, but certainly secondary in its effects on the number of runs scored.
You could try to measure “protection” in real life. Simply compare hitter A hitting in front of a good hitter B versus hitting in front of a average hitter C. I’m sure there are enough natural cases of this happening to be able to at least get in the ballpark estimate of the effects of protection.
Comment by theonemephisto — April 15, 2011 @ 3:20 pm
I think this toolset in general works better in the AL, where you don’t have nearly as steep of drop-offs in batting talent, so the effects of protection/pitching around are minimized. The pitcher just screws everything up.
Comment by theonemephisto — April 15, 2011 @ 3:22 pm
Yup. It’s been done, and they’ve found no substantial evidence it exists.
Of course, a good projection will take account of that sort of issue. Now, if you find manager who magically knows how to move players around appropriately just when they are starting hot and cold streaks, um, yeah, that would be a good idea.
you are assuming the conclusion there.
Number 8 hitters in the NL don’t generally have .400 OBPs or 12.7% BB rates. And further more, you could assume that *some* of the OBP “lost” by Ruiz moving up in the order would be gained by whomever moved down; that is to say – the relative effect on the overall run scoring on any set of lineups is ~constant.
Unless you are suggesting that Carlos Ruiz has some ability to draw walks above and beyond the norm ONLY when batting in front of the pitcher!
The evidence that linear lineup simulations “work” is that that the correlation of simulated games using linear weights in run scored is very high. So high, that any “second order” lineup effects (i.e, protection, batting before the pitcher) are not going to influence the actual conclusion.
You would be better off arguing that psychologically batters do best in whatever batting order position(s) their manager derives from them.
Does anyone more familiar with the math know if a simulation would just be asymptotically equivalent to a Markov approach, anyway?
Comment by Alex Poterack — April 15, 2011 @ 4:29 pm
Simulations that can’t be directly compared to empirical data are hard to prove to be valid enough to trust the results. This is true for all fields. You have to fantastically explicit that the underlying logic (or physics) is accurate and complete. The “and complete” is probably the harder step for any simulation that contains a useful amount of complexity and large numbers of simulators don’t know how to distinguish between “the best we can do right now” and “good enough to generate accurate conclusions”.
That said, baseball is much closer to a state-based system than a lot of other simulations of systems that are being used to come to conclusions even though the simulation cannot be tested for accuracy.
And of course, no team uses one lineup for the entire season anyway. Players move up and down depending on who is starting, who is getting a day off, how a manager feels about a particular batter-pitcher matchup. It is possible that by playing day-to-day matchups, a manger could hit on a more optimal lineup than the optimal lineup generator.
In the NL, the batting orders can change considerably by the middle innings and later as pinch hitters and double switches are used. For this reason, I question whether the magnitude of the lineup optimizing improvement (in terms of projected runs scored) will be overstated, to the extent that the simulation is based on a fixed batting order throughout the game.
A question about those protection simulations – do they find a reduced incidence of intentional walks? Even if a star player hits no better with another good player behind him, doesn’t the fact that he gets more at-bats (and more at-bats in high-leverage situations) mean that protection provides some value?
Simulations account for huge sample sizes (we’re talking hundreds of thousands of instances). Compared to a simulation, a manager has such a tiny amount of games to work with, so the variance will be much higher. If a player happens to homer from the #2 hole when he gets put there, isn’t the manager much more likely to keep him there even if it’s worse off in the long run?
Not necessarily. If you make a simulation environment and then do a train/test approach, at that point you’re mainly capturing the assumptions implicit in the data. Certain categories of naive algorithms in machine learning literally require zero assumptions by the programmer. This doesn’t mean the answers are right, but it means that they’re the result of the data being biased- not the programmer seeing what they wanted to see.
I personally don’t think that naive algorithms are the way to go with baseball, but one could employ some basic rule and physical constraints and then use data to train into them, without introducing much (if any) assumptions. A constrained Hidden Markov Model is a simple example of this sort of thing.
I find it amusing that we’re willing to use simulations to predict nuclear reactions, but balk at the concept of using them for baseball. Methinks one group is underestimating the value of simulation and one group is overestimating it…
Well, it depends what you mean. Almost any simulation is going to be a Markov process (you start it at one state and allow it to run through some path of states, subject to transitions). In that way, there’s definitely a strong connection between the two.
In practice, the approaches are different though. One typical Markov chain approach is to formulate the problem mathematically and then use proofs to determine things about it. Unfortunately, most problems don’t lend themselves to closed form solutions when you do this. You typically can’t do this with a simulation, since simulations usually don’t represent their transition probabilities in a way that you can evaluate them without actually running them (this a variant of the halting problem).
Then you have the more CS approach, where you’d estimate Markov states and transitions from data. This sort of approach IS useful and in theory one could represent any computer simulation using an equivalent Markov chain. With that said, given practical constraints this sort of approach won’t excel at the same things a simulation would. Using a train/test approach, you can get a decent idea of states and state transitions- given the parameterizations of the states you set up (i.e. what they consider). You can estimate these from data, then use the derived weights to simulate or apply math proofs.
A Markov model with the standard form of states/transitions should really be considered one form of representation of a Markov state generator. A typical computer simulation is, in fact, simply another way of representing the same sort of problem. With that said, Markov models tend to require a lot of explicit definition of states- which could quickly get out of hand for a rich model. You could do the same thing a lot more sparsely using a simulation approach.
With that said, I think that both simulation and Markov models are useful tools for looking at baseball. If I had all the time in the world, I’d definitely try to spend some time modeling this sort of stuff. Unfortunately, given that it doesn’t provide a lot of money or social utility (i.e. benefit to the world), I just try to comment on it and hope somebody else has more free time than me :)
Hmmmm, a baseball simulation thread that I somehow missed out on. :)
The way I test mine for accuracy is to compare its results vs those of Vegas. Vegas (closing lines) is pretty smart (smart but still beatable) and is always what any simulator should be first tested against imo. Some in/out of sample testing can protect against back-fitting to a large degree.
Speaking from experience, there is a TON of stuff that goes into a good baseball simulator.