Comments on: When Samples Become Reliable

By: intricatenick

intricatenick — Tue, 21 Jul 2009 23:02:34 +0000

Since you are using terminology such as “stabilize” wouldn’t it be prudent to display a scatter of the correlation vs PAs for all the metrics? Some may not “stabilize” at the number by which i mean the slope of the scatter at the point at which it hits .7 may be important. Of course, this is my attack on everything – “Yes, that may be the case, but what is the derivative doing at that point?”

I’m guessing the marginal benefit of extra plate appearances to the stability would be equal to it’s fraction of the number needed to hit 0.7.

I won’t give too much crap about the 0.7. I do understand that sometimes you have to pick an alpha and stick with it, if only for consistencies sake. For lab standards we’ve gotta go with 0.99

By: Bob

Bob — Wed, 24 Jun 2009 23:48:29 +0000

oops

Those SLGs can’t be true. I will recalculate when I have time.

By: Bob

Bob — Wed, 24 Jun 2009 23:20:17 +0000

I wonder if a similar method can be applied to evaluate a hitter’s PA numbers necessary for “vs a specific pitcher” stats to stabilize. The reason why I am bringing this up is, we often hear announcers and commentators say “this hitter hits well against this pitcher through his career”, and if you look at the actual stats, they are often talking about the sample sizes of 30 or so. That may be laughable to some of us, but maybe not. The way how I see it, THE biggest variable in batting stats for a hitter is the opponent pitcher. If you eliminate that variable, maybe the sample size required for his stats to stabilize become dramatically reduced.

Let me take an example of Eric Chavez against Jamie Moyer. Despite being known to be completely hapless against most other lefties, Chavez somehow managed to hit Moyer to the tune of 323/397/646 in 72 PA. I then did odd-even year split, and found them to be 366/387/766 (31 PA) in the odd years and 285/405/662 (42 PA) in the even years. Except for BAs, which is not surprising, other numbers look pretty good. So, are the OBG and SLG vs a specific pitcher meaningful after 72 PA?

By increasing the number of individual cases (and maybe using Pizza Cutter’s method of splitting into halves), one may be able to determine the sample size needed for “vs a specific pitcher” stats to stabilize. Although some caveats exist (such as these career stats covering many years), this sort of general information could be quite useful.

By: radiosurgery

radiosurgery — Tue, 16 Jun 2009 16:13:49 +0000

Is this not in the glossary? I think this is super useful information, and maybe you guys could either put a link to this article, or just the data in the glossary?

By: CJS

CJS — Sat, 23 May 2009 09:20:56 +0000

I received a brief intro to Bayesian statistics in a class once, and it really clicked for me when I put it into the context of baseball statistics. Basically, I framed it in my mind the way you position what you’ve written here. I don’t remember the specifics of the modeling, but the gist that I do remember is that a guy hitting .350 in April is not a .350 hitter if he hits .275 lifetime. The correct answer is somewhere in between. Pure common sense, but Bayesian statisticians make the effort to build a theoretical framework around the idea, with both subjective and objective (formulaic) underpinnings. This likely isn’t news to you. But your point was interesting to me in that it reminded me of how how baseball brought the abstract theory to life at the time.

By: Pizza Cutter

Pizza Cutter — Sat, 23 May 2009 04:08:55 +0000

Confidence intervals are nice, and they are important. The problem with confidence intervals is the problem of when they cross that line to being “small enough.” There’s not a really good way to tell. Admittedly, my selection of .70 as my cutoff is an arbitrary line in the sand (although I think it has more of a reason than other lines that I might draw.) But decisions are denominated in yes and no. A confidence interval doesn’t give you that.

By: fgelias

fgelias — Sat, 23 May 2009 03:42:19 +0000

cool stuff. i understand why this might be particularly useful for “complicated” statistics (specifically, those with hard to compute variances), but for the simpler rate statistics, what does this tell us that just computing a confidence interval can’t?

for example dan uggla hit 32 HR in 521 AB (for this discussion i ignore the fact that PA!=AB) last season. this is a rate of about 0.06 HR/AB. using the simple formula for the variance of a binary random variable (p*(1-p))/N) i can compute a 95% confidence interval of his “true” rate as (0.040, 0.081).

if i were to use his rate for the current year (7HR in 141AB), the the confidence interval would be a much less informative (0.014, 0.086) because the sample size is so much smaller. doesn’t this provide a much more informative measure of how “reliable” the estimate is?

what additional benefit does a more formal reliability study provide?

By: Scott

Scott — Fri, 22 May 2009 20:43:57 +0000

Cutter,
Because you used PA instead of BIP, it has a quicker, more practical application than it would otherwise. BIP is accessible, but junkie baseball fanatics know how many PA’s batters have off the top of their head, or at least can estimate. I for one am glad you used PA.

By: Pizza Cutter

Pizza Cutter — Fri, 22 May 2009 20:20:18 +0000

I’ve actually done some similar work looking at those sorts of trend lines. Worth a shot in this case.

By: Peter Jensen

Peter Jensen — Fri, 22 May 2009 19:52:18 +0000

Pizza – Wouldn’t time series analysis be one way to approach a problem like Coco’s walk rate? Split his career into 200 PA segments and measure his walk rate in each segment and the standard deviation between segments and then test for a relationship varying with time?