The Observer Effect and Fan Projections

In high school, I — like many — was taught that the Heisenberg uncertainty principle refers to changes that the act of observing can have on the phenomenon being observed. So, for example, if I (for some unknown reason) were attempting to observe a particle and was using some sort of device that would bounce a photon off of said particle, I would inevitably change the particle’s position or momentum. Thus, in attempting to measure it, I would’ve have actually changed it. To measure it au naturale would, in this incredibly hypothetical scenario, be impossible.

Despite the fact that my high school was impossibly prestigious and reputable, it turns out my teacher only had it sorta right. The actual definition of Heisenberg’s uncertainty principle (per Wikipedia, which is always right) is that “certain pairs of physical properties, like position and momentum, cannot both be known to arbitrary precision. That is, the more precisely one property is known, the less precisely the other can be known.”

What my physics teacher was describing is actually known as the Observer Effect. And while (again, per Wikipedia) the two are related — Heisenberg was integral in defining the Observer Effect, too — it’s usually the Observer Effect that people are meaning when they say Heisenberg uncertainty principle.

And it was the possible influence of the Observer Effect that I considered when FanGraphs Overlord David Appelman announced the arrival of Fan Projections this past weekend.

As as been noted in these electronic pages — most recently by Dave Cameron — the idea of Fan Projections/Community Forecasting relies on the Wisdom of the Crowd, a concept ably explored by James Surowiecki in a book of the same name. That book begins with an anecdote about Statistical Wunderkind and Pretty Racist Francis Galton. The story (dramatized by Radio Lab here) goes like this: Galton visits a country fair. At the fair, he comes across a weight-judging competition, where fair-goers are encouraged to guess the weight of an ox. No one guesses the weight of the ox exactly, but Galton, gaining permission from the organizers to do so, collects all 787 entries and finds that the “average” guess comes out to 1,197 pounds. The actual weight of the ox? 1,198 pounds — i.e. closer than every single one of the entries.

That’s an exciting discovery. But the difference between Galton’s experiment and the one we’re conducting here is that the Crowd (read: you and you and you) can see the results in real time. If each of the fair-goers from the Galton anecdote had been able to see the guesses that had preceded their own, would that have affected the results? If you and you and you can see Derek Jeter‘s current projection before making your own, how will that affect your own assessment of Jeter?

In other words: Will the Observer Effect come into play here?

The answer is: I don’t know. Of course, there’s a big emphasis on the “I” there. Here are some subjects in which I have little in the way of expertise: physics, statistics, personal hygiene. (The last of these, I recognize, isn’t wholly relevant to the present conversation; still, it’s the truth.) But you know who might know? People who are smarter than me. People like Tom Tango and Dave Cameron. So I asked them. Below are their respective responses.

Here’s Tom Tango:

No, I don’t think there will be a bias caused by seeing other fans’ forecasts. You will note that David shows the selections a bit differently from the standard line. Furthermore, fans are notoriously stubborn.

Having said that, it’s a simple matter to look at the forecasts early on and later on and seeing if the standard deviations of the selections are tighter the later the selections. I’d be shocked if you find anything.

Personally, I’d be shocked if I found anything, too — but only because that would mean that I’d checked.

And now Dave Cameron:

I think it’ a legitimate issue, but David has designed the inputs in such a way that limit the opportunity to just repeat what has already been done. Since most of the rate stats are calculations based on the raw inputs, the fans aren’t going in and just projecting players for the .360 wOBA that everyone else already concluded was likely. Instead, they are projecting the components, which are less likely to be observed. I know that I can tell you what the wOBA/WAR projections are currently at for a few players, but I have no idea how many doubles and triples anyone is projected to hit.

I think that will mitigate some of the problems the observer effect raises. We can’t eliminate it, of course, but it shouldn’t be a big enough problem to ruin the projections.

So the consensus here seems to be that no, mostly likely the Observer Effect will not influence (or noticeably influence) the Fan Projections. Consensus is different than fact, of course, but it’s good enough for now.



Print This Post



Carson Cistulli has just published a book of aphorisms called Spirited Ejaculations of a New Enthusiast.


Sort by:   newest | oldest | most voted
Tony Gwynn
Guest
Tony Gwynn
6 years 7 months ago

The Galton story is apocryphal.

divakar
Guest
divakar
6 years 7 months ago

Carson, I hope I’m not being a pain in the ass by writing this, but I think you misused the Observer Effect in trying to point out a valid issue with the Fan Projections.

The Obesrver Effect is truly fascinating in physics… One of those things that make you realize our physical world is not as it seems…

In fan projections, however, I think you’re talking about something else – I think in this context, you’re simply describing bias. I don’t think they are similar at all. Tango even said “bias” in his response. If people see other people’s projections, they might be biased by them in producing their own. That seems reasonable enough, and I’m not sure why Tango thinks fans are so stubborn as to NOT be biased. I like Cameron’s answer better, as perhaps the Fan Projection methodology will prevent bias from altering the projections…

To me the observer effect might be noticeable in baseball if you talk about Brian Bannister or something. I mean, we observe him (and measure his ground-ball rate) and he changes as a result. Thus, our previous measurements of him are now inaccurate (simply as a by-product of measuring him at all).

Even this, however, is a stretch. In Fantasyland, Sam Walker talks about the Jacque Jones situation: Jacque Jones is incredulous that batting .300 doesn’t mean much. It’s not like our observations of him magically transformed him into a hitter who focuses on OBP instead.

So in this case, the “observer effect” is really just someone else taking criticism well. Unfortunately, I don’t think we can say the same thing for electrons.

TomG
Member
TomG
6 years 7 months ago

divakar is correct in that this isn’t the observer effect as much as it is bias, however, it would be fun to imagine that the observer effect could come into play with the fan projections. Say the fans, in taking a measurement on a current free agent’s future value (i.e. projecting his future production), affect the player’s current financial value (since free agents are theoretically paid in part for future production), provided there is a GM out there that lends even the tiniest bit of credence to the wisdom of the crowds.

In this scenario, it’d be possible for us Fangraphers to potentially drive up a player’s value, thus possibly affecting his financial windfall. What I propose to test this hypothesis is that we all project Bobby Crosby to hit an absurd .300/.375/.500 and then watch as Dayton Moore signs him to a 5-year, $36MM contract.

mystlyfe
Guest
mystlyfe
6 years 7 months ago

Agreed, it was not proper usage of the term “observer effect.” Glad someone else beat me to the punch, my physics knowledge was crying out in pain.

tangotiger
Guest
tangotiger
6 years 7 months ago

David said it clearer, but I gave the same answer:

“You will note that David shows the selections a bit differently from the standard line.”

I get the same question asked about my Fans Scouting Report and people being biased by knowing their UZR. I don’t ask them to evaluate their UZR, but their arm strength and foot speed. Sure, the UZR could still bias that, but, before this year, there were no UZR for rookies, and rookie evaluations still came out looking fine.

divakar
Guest
divakar
6 years 7 months ago

fair enough

Nathaniel Dawson
Guest
Nathaniel Dawson
6 years 7 months ago

The Subject-expectancy effect, is a form of reactivity that occurs in scientific experiment or medical treatment when a research subject or patient expects a given result and therefore unconsciously affects the outcome, or reports the expected result.

http://en.wikipedia.org/wiki/Subject-expectancy_effect

Rick
Guest
6 years 7 months ago

What will be interesting is to compare the predictive accuracy of the projections made by all fans compared to those made by fans of the team on which the player played the prior year. Does watching the player more and presumably knowing more than the average baseball fan lead to more accurate predictions or merely more optimistic ones?

oldjacket
Member
oldjacket
6 years 7 months ago

There will probably be an opitmism bias. Fans are more likely to project players they root for, and have hope for. I bet these come out more optimistic than the Bill James projections.

Brad Johnson
Member
Member
6 years 7 months ago

Hence why I will be projecting the Mets before my home town Phillies.

jthomas
Guest
jthomas
6 years 7 months ago

Whether apocryphal or not, you’ve missed an aspect of the Galton story – the guesses came from non-experts, common folk at a fair. The takeaway is not that the median was closer than any individual gift, the takeaway was that the collective managed to be quite accurate yet culled from non-experts.

To piggyback on Rick, another invocation of the Galton story would be suggesting the projections be culled exclusively from those who do NOT follow the players regularly.

Brad Johnson
Member
Member
6 years 7 months ago

Interesting and unrelated factoid of the day: Wikipedia is considered to be more accurate than the Encyclopedia Britanica (which is supposed to be the next most accurate).

Although it was 2007 or so when I actually first read that so who knows if it still applies.

Bronnt
Member
Bronnt
6 years 7 months ago

Maybe you should check wikipedia

BA Baracus
Guest
BA Baracus
6 years 7 months ago

If you have a large number of projections on a single player, the bell curve will take care of any outliers. Hence the over optimistic homer and the over pessimistic hater will cancel each other out or not have much of an effect on the aggregate results. I think its a brilliant idea and am thinking about using the fans projections for my own cheat sheets. Provided that enough projections have been accumulated and nothing major happens in spring training.

Bronnt
Member
Bronnt
6 years 7 months ago

This isn’t a purely random sample, though. You have volunteer bias-meaning that people who come in to project are those who have something to say. I would assume that there’s not a large sample of fans who were just waiting for project a .200/.200/.300 line for A-Rod, but there’s plenty of Yankee and A-Rod fans who simply want to project him and are generally optimistic.

If you wanted to make the sample random, perhaps you could make every visitor of the site project 10 players before they were allowed to view content-but of course, stuff like that is lame and ruins the interwebs!!1

jbluestone
Member
jbluestone
6 years 7 months ago

Just saw this, I wrote something similar down below.

Jimbo
Member
Jimbo
6 years 7 months ago

In our 24-hour news channel society, patience isn’t the trendy thing. I have to keep reminding myself that the benefits of this effort probably won’t be known (quantifiably) for a year or more.

Eventually, there will be all sorts of potential for dissecting the inputs and gaining insight into the real wisdom of this baseball crowd! I imagine this is how everyone felt at the start of the Human Genome Project. ;-)

Cody
Guest
Cody
6 years 7 months ago

Socioeconomics refers to this phenomenon as Anchoring, as described under the Anchoring and Adjustment Heuristic heading here: http://en.wikipedia.org/wiki/Anchoring

Wexler
Guest
Wexler
6 years 7 months ago

Whatever the semantics, I thought this was a really perceptive article. Enjoyed it.

jbluestone
Member
jbluestone
6 years 7 months ago

I feel like there is a “participation bias” though

My predicition is that the players who are projected will have significantly better projections then most systems, because people are “choosing” which players to pick, which means they will predict the players they are most interested in. Your 3 per day focus will help with some of that, but the fact remains, that most people will probably project their favorite players, or the players they have kept on their fantasy teams, etc.

Therefore you will have a participation bias. Defined as:

Participation bias is a phenomenon in which the results of elections, studies, polls, etc. become non-representative because the participants disproportionately possess certain traits which affect the outcome.

For example, people calling in to comment on a radio talk show tend to be people who feel very strongly one way or the other, it doesnt represent the people who do not have strong feelings.

Either way, I love the idea, but I suspect that without more of a push from you guys to get a random sampling of projections from the fans, it will have a form of selection bias.

Slacker George
Member
Member
Slacker George
6 years 7 months ago

I’m a 2 on the scale of 1-10 as far as statistical analysis knowledge, so take my suggestion with a grain of salt and a dash of hoagie oil:

Why not create a control group of players and withhold the current fan projections until the end date? Therefore, no bias for the control group. Compare the control group with the others and measure all the relevant statistical measures.

That would give you a better idea of whether the bias is non-trivial, right?

Steven J. Sanders III
Member
Steven J. Sanders III
6 years 7 months ago

Isn’t one of the the main principles of a wise crowd the independence of opinion? It seems like it would be an issue because of the possibility for a groupthink-type result because each individual in the crowd can what the prediction is in real-time. Per Dave Cameron’s note above, I suppose they’ve already attempted to address this. I also wonder if there is some Hawthorne-type effect here? If people know they are participating in the study, does that impact their behavior? If so, will there be a truly diverse set of opinions?

Anyway, this is probably all moot, right? Anyone who is smart enough to have built this site from the ground up probably thought of a contingency for anything I might mention in a comment…

wpDiscuz