No, I didn’t mention that but it’s a good point. Of the 34 CLIFFORD candidates, only 3 were also identified by Marcel. So, yes, there may be something there to investigate. I just haven’t gotten back to it. So, not completely a loser, but still thought the overall “failure” story might be worth telling.
Result fabrication leads to much more interesting conclusions than “I have found nothing interesting.” (You can also run correlations of enough data to get a nice positive result; did you know that sales of organic foods have a brilliantly high correlation to autism diagnoses?)
Applying effort to find the right answer is hardly going to get you a grant. As it stands, Lennay Kekua does not like your article.
Comment by John R. Mayne — January 23, 2013 @ 10:20 am
Wow, only 3? That’s very interesting, and definitely makes me think it wasn’t a failure at all. If you can identify an entirely different set of players poised to decline, then that’s fantastic!
Based on what you’ve said, if you create a new metric that adds together all of the CLIFFORD candidates and all the Marcel candidates, then this new metric would be much, much better than Marcel. You haven’t failed at all.
To be more elegant about it, of course, you’ll want to see how Marcel is identifying these candidates and incorporate it into your own metric. Or, perhaps more accurately, projection systems like Marcel will eventually want to incorporate your new findings, I imagine.
@Matt and @cass: I totally agree–for this I wanted to make sure I reported out on one type of failure (re: the hypothesis that relative risk would be better with CLIFFORD than Marcel), but there is likely something to be had by combining the two together.
A lot of your CLIFFORD factors seem common sense as well.
In terms of future progress:
– Did you re-run the relative risk including the 5th parameter, change in K%? That might boost your relative risk even higher. It’s probably correlated with contact% but unless that correlation is perfect you’re likely losing some predictive power.
– Have you considered including more years in your data set?
– I’m somewhat new to Sabermetrics so perhaps this has been answered before, but is there a reason you’ve not built a linear regression model with these data? Wouldn’t this help you refine your model even further?
No adjustments were made in CLIFFORD. So Marcel is likely picking up the “natural” decline candidates given aging curves, etc, whereas CLIFFORD is identifying candidates based on other changes to their peripherals.
i wouldn’t give up just yet either. I would want to know A) of the differences between clifford and marcel, which has a higher success rate? B) if clifford isn’t taking age factors into account, do you surpass marcel once you do add these factors in? you use UBR and SPD to account for aging but i’m not sure that’s sufficient unless changes in those stats from year to year are largely attributable exclusively to age. C) what happens if you expand the categories?
If you feel like Marcel effectively disproved your hypothesis and that this research doesn’t warrant further pursuit, i don’t think you proved that in this article.
I love the “inside baseball analytics” stuff, but i see this more as “showing your work” than “acknowledging a complete bust.” Maybe this could be part 1 in a series about developing a better prediction? Make some adjustments and report back, etc. Even if clifford doesn’t pan out, it could be used to test against existing projection systems and compare how, say, marcel does at predicting decline vs other systems
Comment by tylersnotes — January 23, 2013 @ 1:41 pm
And i’m sure Bill, that your peripherals in question are often associated with the “natural” declines/trends per Marcel’s system – i mean even the KPI’s you referenced: z-contact, UBR, speed, & FA/catching up with the FA, etc. all usually decline with age (at least that’s a good assumption). I think if you can disassociate variables with age trends, your research would have significant utility. I think it still has great significance because even accounting for age, a combination of certain variables will associate moreso with a visible decline.
With that said, your system should still show value if it points to red flags that dont often correlate significantly with age trends, but it’s incestual if the relative risks you start with all relate to age.
I always enjoy when someone works relentlessly to wind up going against their own hypothesis. It shows integrity.
Comment by rotobanter — January 23, 2013 @ 3:13 pm
It seems like even if your decline predictions had already been found by Marcel, that doesn’t make your result a negative one. It still would have been a positive result (you confirmed your hypothesis), it’s merely that you would have replicated the positive result found by Marcel.
For starters, we don’t do enough of this. By we, I mean any researcher in just about any discipline–not just baseball analysis. More and more, journals, newspapers, online forums, and conferences are filled with the reporting out of positive results (“hey, look, I confirmed my hypothesis! I discovered a new thing!”). And while positive results are interesting and important in their own right, negative results are (or, should be) just as important. Any discipline progresses in large part by falsifying hypotheses, replicating results, and figuring out what doesn’t work so it can focus on what might.
This is called “Reporting bias” and is a huge problem in all fields that I am familiar with. Boring negative results don’t get reported and when they do, they don’t get the press releases or the headlines in the lay press that the sexy positive ones do. You could have 14 people try to prove a hypothesis and fail, then one “proves” it correct with a p value of 0.1 and guess which one people remember? Now you have to do the original 14 studies over again as “confirmatory” studies and then they get reported. It’s a waste of resources.
“I have not failed. I’ve just found 10,000 ways that won’t work.” -Thomas Edison
Comment by GordieDougie — January 23, 2013 @ 5:34 pm
Yeah, I would think Marcel is just picking the low-hanging fruit (predicting regression for guys coming off ridiculous years), whereas your system clearly is not doing so, but rather is filtering for guys who are coming off a relatively bad year. Look forward to more info.