Comment by Tony Gwynn — December 3, 2009 @ 4:30 pm
Carson, I hope I’m not being a pain in the ass by writing this, but I think you misused the Observer Effect in trying to point out a valid issue with the Fan Projections.
The Obesrver Effect is truly fascinating in physics… One of those things that make you realize our physical world is not as it seems…
In fan projections, however, I think you’re talking about something else – I think in this context, you’re simply describing bias. I don’t think they are similar at all. Tango even said “bias” in his response. If people see other people’s projections, they might be biased by them in producing their own. That seems reasonable enough, and I’m not sure why Tango thinks fans are so stubborn as to NOT be biased. I like Cameron’s answer better, as perhaps the Fan Projection methodology will prevent bias from altering the projections…
To me the observer effect might be noticeable in baseball if you talk about Brian Bannister or something. I mean, we observe him (and measure his ground-ball rate) and he changes as a result. Thus, our previous measurements of him are now inaccurate (simply as a by-product of measuring him at all).
Even this, however, is a stretch. In Fantasyland, Sam Walker talks about the Jacque Jones situation: Jacque Jones is incredulous that batting .300 doesn’t mean much. It’s not like our observations of him magically transformed him into a hitter who focuses on OBP instead.
So in this case, the “observer effect” is really just someone else taking criticism well. Unfortunately, I don’t think we can say the same thing for electrons.
The Subject-expectancy effect, is a form of reactivity that occurs in scientific experiment or medical treatment when a research subject or patient expects a given result and therefore unconsciously affects the outcome, or reports the expected result.
Comment by Nathaniel Dawson — December 3, 2009 @ 4:32 pm
What will be interesting is to compare the predictive accuracy of the projections made by all fans compared to those made by fans of the team on which the player played the prior year. Does watching the player more and presumably knowing more than the average baseball fan lead to more accurate predictions or merely more optimistic ones?
divakar is correct in that this isn’t the observer effect as much as it is bias, however, it would be fun to imagine that the observer effect could come into play with the fan projections. Say the fans, in taking a measurement on a current free agent’s future value (i.e. projecting his future production), affect the player’s current financial value (since free agents are theoretically paid in part for future production), provided there is a GM out there that lends even the tiniest bit of credence to the wisdom of the crowds.
In this scenario, it’d be possible for us Fangraphers to potentially drive up a player’s value, thus possibly affecting his financial windfall. What I propose to test this hypothesis is that we all project Bobby Crosby to hit an absurd .300/.375/.500 and then watch as Dayton Moore signs him to a 5-year, $36MM contract.
You’re right, Divakar. Electrons take criticism horribly.
Also, “bias” seems like a reasonable term to use. Perhaps the metaphor I’m proposing doesn’t work quite right. Certainly, in the Brian Bannister case that you present, Bannister is the particle and FanGraphs is the photon-shooting device. Bannister changes because he’s being measured.
Maybe the term “Observer Effect” is used more broadly (read: imprecisely) in popular culture? More than through physics, that’s how I’ve been exposed to it. In any case, it’s probably just synonymous with bias.
Comment by Carson Cistulli — December 3, 2009 @ 4:59 pm
There will probably be an opitmism bias. Fans are more likely to project players they root for, and have hope for. I bet these come out more optimistic than the Bill James projections.
Whether apocryphal or not, you’ve missed an aspect of the Galton story – the guesses came from non-experts, common folk at a fair. The takeaway is not that the median was closer than any individual gift, the takeaway was that the collective managed to be quite accurate yet culled from non-experts.
To piggyback on Rick, another invocation of the Galton story would be suggesting the projections be culled exclusively from those who do NOT follow the players regularly.
If you have a large number of projections on a single player, the bell curve will take care of any outliers. Hence the over optimistic homer and the over pessimistic hater will cancel each other out or not have much of an effect on the aggregate results. I think its a brilliant idea and am thinking about using the fans projections for my own cheat sheets. Provided that enough projections have been accumulated and nothing major happens in spring training.
Comment by BA Baracus — December 3, 2009 @ 5:47 pm
This isn’t a purely random sample, though. You have volunteer bias-meaning that people who come in to project are those who have something to say. I would assume that there’s not a large sample of fans who were just waiting for project a .200/.200/.300 line for A-Rod, but there’s plenty of Yankee and A-Rod fans who simply want to project him and are generally optimistic.
If you wanted to make the sample random, perhaps you could make every visitor of the site project 10 players before they were allowed to view content-but of course, stuff like that is lame and ruins the interwebs!!1
In our 24-hour news channel society, patience isn’t the trendy thing. I have to keep reminding myself that the benefits of this effort probably won’t be known (quantifiably) for a year or more.
Eventually, there will be all sorts of potential for dissecting the inputs and gaining insight into the real wisdom of this baseball crowd! I imagine this is how everyone felt at the start of the Human Genome Project. ;-)
David said it clearer, but I gave the same answer:
“You will note that David shows the selections a bit differently from the standard line.”
I get the same question asked about my Fans Scouting Report and people being biased by knowing their UZR. I don’t ask them to evaluate their UZR, but their arm strength and foot speed. Sure, the UZR could still bias that, but, before this year, there were no UZR for rookies, and rookie evaluations still came out looking fine.
Comment by tangotiger — December 4, 2009 @ 10:31 am
I feel like there is a “participation bias” though
My predicition is that the players who are projected will have significantly better projections then most systems, because people are “choosing” which players to pick, which means they will predict the players they are most interested in. Your 3 per day focus will help with some of that, but the fact remains, that most people will probably project their favorite players, or the players they have kept on their fantasy teams, etc.
Therefore you will have a participation bias. Defined as:
Participation bias is a phenomenon in which the results of elections, studies, polls, etc. become non-representative because the participants disproportionately possess certain traits which affect the outcome.
For example, people calling in to comment on a radio talk show tend to be people who feel very strongly one way or the other, it doesnt represent the people who do not have strong feelings.
Either way, I love the idea, but I suspect that without more of a push from you guys to get a random sampling of projections from the fans, it will have a form of selection bias.
Comment by jbluestone — December 4, 2009 @ 12:34 pm
Just saw this, I wrote something similar down below.
Comment by jbluestone — December 4, 2009 @ 12:35 pm
I’m a 2 on the scale of 1-10 as far as statistical analysis knowledge, so take my suggestion with a grain of salt and a dash of hoagie oil:
Why not create a control group of players and withhold the current fan projections until the end date? Therefore, no bias for the control group. Compare the control group with the others and measure all the relevant statistical measures.
That would give you a better idea of whether the bias is non-trivial, right?
Comment by Slacker George — December 4, 2009 @ 6:19 pm
Isn’t one of the the main principles of a wise crowd the independence of opinion? It seems like it would be an issue because of the possibility for a groupthink-type result because each individual in the crowd can what the prediction is in real-time. Per Dave Cameron’s note above, I suppose they’ve already attempted to address this. I also wonder if there is some Hawthorne-type effect here? If people know they are participating in the study, does that impact their behavior? If so, will there be a truly diverse set of opinions?
Anyway, this is probably all moot, right? Anyone who is smart enough to have built this site from the ground up probably thought of a contingency for anything I might mention in a comment…
Comment by Steven J. Sanders III — December 5, 2009 @ 10:11 am