# Don’t Trust Stats This Week

The absolute hardest part of being a fan or analyst is avoiding the use of small samples to make a definitive claim. Allowing unreliable information to shape an opinion or serve as the foundation for an important decision is a mistake. This especially occurs in the opening week of the season when the statistics produced are meaningless and everyone is susceptible to the manipulation of numbers out of sheer joy that baseball has resumed. Just because Ramon Hernandez went 4-5 yesterday does not mean he has regained his stroke and will have an all-star caliber season. The same can be said if he starts the season 12-20, or 21-40.

We have enough information about his true level of abilities at this point in time that 20 trips to the dish is nothing more than a mere blip in the dataset. But when can we be sure that a trend to open the season is actually indicative of a noteworthy change?

The answer is not cut and dried. It depends upon the specific statistic being examined. As I wrote back at the beginning of 2009, statistics “stabilize” at different thresholds of plate appearances. A trend can be considered significant for certain statistics sooner than it can for others. Having this knowledge can prevent rash decisions in fantasy leagues, and can aid in the avoidance of anointing players as having breakout years or writing them off completely. The results in the prior article were based upon the tremendous work of Russell Carleton, my former colleague at Statistically Speaking, and current analyst for the Cleveland Indians.

The method known as split-half reliability was utilized, which measures the correlation between different parts of the same dataset. An example would involve separating Matt Holliday‘s even-numbered plate appearances from his odd ones, and then running a correlation on both bins. When the correlation between the bins is somewhere around the 0.7 range–correlations run from -1 to 1, with +/- 0.7 or above indicating significance in a statistical study–the statistic can be considered useful in forming opinions and noting trends.

By ‘useful’ I am referring to the notion that our expectations moving forward can be more narrowly defined. The goal then becomes finding the lowest PA total at which point the correlation is significant. Holliday has averaged around an 11% walk rate over the last three seasons. If in his first 200 PAs this season his walk rate has dropped to five percent, the split-half reliability test will tell us how closely his second 200 PA bucket will mirror the first.

If the correlation is close to 1.0 then we can say with an increased level of confidence that his walk rate will remain at five percent. Even though his past exploits are known, that specific statistic can be indicative of a change at around the 200 PA mark. When using numbers to form opinions, isn’t that type of assumed reliability the desired result? Nobody knows for sure what exactly will happen over those next 200 PAs, but tests like this help to reduce the range of possibilities and eliminate some guessing in the dark.

The thresholds for various statistics offered on this site are as follows:

``` 50 PA: Swing %
100 PA: Contact Rate
150 PA: Strikeout Rate, Line Drive Rate, Pitches/PA
200 PA: Walk Rate, Groundball Rate, GB/FB
250 PA: Flyball Rate
300 PA: Home Run Rate, HR/FB
500 PA: OBP, SLG, OPS, 1B Rate, Popup Rate
550 PA: ISO```

How can this information be used? Well, it’s unlikely that anyone will rack up 50 plate appearances in the first week of the season, so it will take at least two weeks before the first trend on offense can truly be tracked. A hot start from an unexpected player might be interesting to flag for future evaluation, but cannot be considered noteworthy until much more of the season has passed.

My goal isn’t to serve as a wet blanket or Debbie Downer, but rather to shed light on seminal research that can help everyone keep some perspective in the opening week(s) of the season. Not only should fans and analysts avoid letting preformed opinions shape their perceptions of a player based on numbers produced at the start of the season, but the statistics themselves should not even be considered worth discussing until they surpass the aforementioned plate appearance marks.

Print This Post

Eric is an accountant and statistical analyst from Philadelphia. He also covers the Phillies at Phillies Nation and can be found here on Twitter.

Guest
CircleChange11
5 years 5 months ago

Just because Ramon Hernandez went 4-5 yesterday does not mean he has regained his stroke and will have an all-star caliber season.

But, his is clutch, right? Heh Heh.

Good on ya for posting the thresholds. Useful information.

Guest
Joe
5 years 5 months ago

Good article.

“Wet blanket” and “seminal research” shouldn’t be used in the same sentence, though…

Guest
5 years 5 months ago

They don’t call me the pun king for nothing!

Guest
B N
5 years 5 months ago

Calm it down there, Pun King Brewster.

Guest
MikeS
5 years 5 months ago

You mean I can’t extrapolate Ramon Hernandez’s 0.3 WAR day into a 48.6 WAR season? Time to dump him from my fantasy team. Maybe I can trade him for that Pujols guy and his wRC+ of -100.

Guest
5 years 5 months ago

If Kenny Mayne were still on Sportscenter he would’ve noted that Heyward and Hernandez were on pace for 162 HR, but not in the same game, as that would be a record.

Member
5 years 5 months ago

Hat tip to me on that comment!

Guest
5 years 5 months ago

My bad, Alex, yeah, the Remington-meister reminded me of that old Mayne gem.

Guest
CircleChange11
5 years 5 months ago

I remember 87 when George (or Jorge) Bell started off with a 3-HR game (and Karl ‘Tuffy’ Rhodes did the same thing for the Cubs, vs. Dwight Gooden, no less) … and the “on pace for 486 HR” jokes were rampant … which would, actually, be some kind of a record.

Guest
Cabrone
5 years 5 months ago

Great exercise, can you rerun it for pitchers, or are do you think the numbers would be about the same?

Guest
Mile
5 years 5 months ago

I just learned about split half reliabaility the other day, so this is especially awesome. Is there any difference between career trends and in season trends? If so, did you find thresholds for them?

Guest
dutchbrowncoat
5 years 5 months ago

april fools! these stats are actually useful and albert pujols will be the worst hitter in baseball this year!

Member
Cidron
5 years 5 months ago

yeah, jus dumped that pujols guy yesterday. All he does is hit into double plays.

Guest
powder blues
5 years 5 months ago

A uniquely Fangraphs article – love it.

Guest
B N
5 years 5 months ago

Hmm… looks like there’s an argument brewing between Seidman and Appelman:
http://www.fangraphs.com/blogs/index.php/2011-stats-working/

So are they working or should we not trust them?! Inquiring minds want to know! (For the record, I side with Seidman. I’m pretty sure that Kimbrell won’t put up a -2.28 ERA or FIP for the balance of the season. … Though I’ve been wrong before. Maybe the man just erases runs!)

Guest
phoenix2042
5 years 5 months ago

thats going to be really helpful for fantasy managers. And i was convinced the nats werent going to score a run all season, darn. And it kind of looked like matt kemp was going to take 486 walks this (3 per game)!

Guest
5 years 5 months ago

Amusing that this is posted right after “2011 Stats Working!”

As an English major, I’m naturally a bit unsure of what I’m seeing, though. Which stats are for pitchers, which for hitters? Is there a difference between the thresholds for similar statistics whether they’re for pitchers or hitters? Like, say, strikeout rate?

Guest
5 years 5 months ago

This is just for offense. Sorry for the confusion.

Guest
5 years 5 months ago

No worries. Thanks for clearing that up!

Member
gnomez
5 years 5 months ago

Is there any available data on when stats stabilize for pitchers?

Guest
Kapellmeisters
5 years 5 months ago

Great stuff here

Guest
phoenix2042
5 years 5 months ago

when does pitch velocity become significant?

Guest
pedro
5 years 5 months ago

The moment the ball hits you in the ear.

Guest
Pizza Cutter
5 years 5 months ago
Guest
5 years 5 months ago

Yeah, the big thing with pitchers is that around 150 PA we gain some idea about the change in groundball rate. This was helpful in identifying Cliff Lee’s breakout a few years ago.

Guest
Jeremy
5 years 5 months ago

How many PA before BABIP will stabilize?

Guest
5 years 5 months ago

Unfortunately, A LOT.

Guest
Pizza Cutter
5 years 5 months ago
Guest
Jeremy
5 years 5 months ago

Very cool. Do you think it’s around the same amount of balls in play for hitters BABIP too? So you’re looking at a sample time of 7-8 years?

Member
Member
dudley
5 years 5 months ago

we might not be able to trust the stats, but we can trust our eyes, right? e.g., nagging injuries don’t seem to be sapping braun’s power anymore.

Member
Member
5 years 5 months ago

We can begin trusting them next week right?

Guest
5 years 5 months ago

Swing Rate baby!!!

Guest
5 years 5 months ago

Could be longer. Austin Jackson (since he is the poster-boy for the dangers of babip) had 675 plate appearances last year, and put about 450 balls in play. At that rate, it would take him 8-9 years to reach 3800 balls in play.

What this says to me is that babip never truly stabilizes, because a player won’t be the same at the end of that eight years that he was at the beginning. It takes you eight years of data before you can say “ok, I have a good idea of this guy’s true talent.” Only, you can’t say that, because what you know is his average over that eight year span. He might not be that player anymore.

What I would be interested in seeing is how long it takes for xbabip to stabilize.

Guest
CircleChange11
5 years 5 months ago

And players with horrible BABIP stats won;t be in th eleague that long.

So, just like pitchers, whatr we get are comparing BABIPs of guys that are in the league for 10 years … so we’re trying to find differences in the top 5% of an already elite field.

Not surprisingly we come to the conclusion that “there’s no real difference”.

Member
mcneo
5 years 5 months ago

I can just see it now…

Cardinals Announcer:
“Tony La Russa is now pinch hitting Chris Carpenter for Albert Pujols because Chris Carpenter has a .333 BA in even numbered at bats with a slug percentage of 1200!! Albert is about to have an odd numbered at bat and he only hits .311/.357/.516 in odd numbered at bats… “

Guest
Detroit Michael
5 years 5 months ago

Do you no longer write for BaseballProspectus.com? I don’t recall their regular contributors usually writing for multiple websites (not that I mind).

Guest
5 years 5 months ago

Correct. As of this past Monday I am exclusive to Fangraphs and my personal site, BrotherlyGlove.com.

Guest
5 years 5 months ago

Wait, so you’re saying I shouldn’t have dropped Albert Pujols after his three double play performance yesterday???

Guest
5 years 5 months ago

My advice would be to hold onto him. Give him at least a week.

Guest
don
5 years 5 months ago

ChiSox are going to score 5000 runs this year.

Guest
woodman
5 years 5 months ago

J.P. Arencibia is now on pace for 324 homers. The Blue Jays are on pace for 648 homers. Sounds reasonable.

Guest
pft
5 years 5 months ago

SSS stats are simply small pieces of new information that if used wisely may be useful in some cases. Any SSS stats must be coupled with observation to provide any meaning.

SSS stats are not predictive, however, unusually good or bad numbers could signal a break out year or collapse, or suggest a hidden injury (or be evidence a players recovery from a previous injury is not complete) .

In any event, while not predictive, the stats are a good indicator of how a player has performed over a short period, even if some of that may be due to good or bad luck. We get excited over no hitters and batters hitting for the cycle, and these are SSS stats in the extreme, and influenced by luck..

Guest
Kevin
5 years 5 months ago

Another joke about how I shouldn’t judge Albert Pujols this year based on his performance yesterday.