Measurement and Observation Alters the System: What Does this Mean for Social Science Research?
When we read research papers or even the published results of surveys, we need to critically examine the samples, sample sizes, and statistical techniques used. We also need to understand the author/organization to understand what biases and motivations may influence the study/survey. Critical thinking skills are critical.
“In physics, the observer effect is the disturbance of an observed system by the act of observation. This is often the result of utilizing instruments that, by necessity, alter the state of what they measure in some manner.” https://en.wikipedia.org/wiki/Observer_effect_(physics)
While the quote above is from Wikipedia, which is not a scholarly citation, it is accurate. I learned the concept in college physics, and it is one concept that stayed with me these 40 plus years. The Hawthorne Effect, where workers responded to changes, regardless of the change, since they knew they were being watched, also showed this concept. I thought about it often over the course of two doctoral programs in the social sciences. Many research papers use surveys and interviews and take measurements. Many then use statistical measures to analyze the data and draw conclusions. I recall some papers where half the content developed arcane models and then used them to torture the data.
One more physics concept that may be relative to the observer effect. The Heisenberg Uncertainty Principle states we may measure either the location or the momentum of a particle, but not at the same time. The Stanford Encyclopedia of Philosophy states:
“The notion of “uncertainty” occurs in several different meanings in the physical literature. It may refer to a lack of knowledge of a quantity by an observer, or to the experimental inaccuracy with which a quantity is measured, or to some ambiguity in the definition of a quantity, or to a statistical spread in an ensemble of similarly prepared systems.”
The measurement and uncertainty principles relate directly to both qualitative and quantitative analysis in the social sciences.
- Measuring and observation changes behavior. Think about the surveys you have taken. Did you answer a bit differently than you would honestly answer because you thought you understood the survey purpose or the author’s objectives? I know I do. Likewise, do you change your behavior when you are observed? I suppose the law of large numbers would mitigate this problem, but often sample sizes are just large enough to meet statistical relevancy requirements.
- Sample sizes are often skewed. Most samples are not truly random, yet study authors often use stochastic statistical analysis tools. I suspect most study samples are more clustered sampling than random sampling. Using the wrong statistical tool may taint the analysis.
- Samples may not be generalizable, regardless of the technical statistics requirements. For example, taking a sample from a single school or even a school district and then using the stochastic tools to derive conclusions on the sample and generalizing them to all schools or school districts could provide misleading conclusions. For example, I saw papers in my doctoral work that did exactly this with schools. I also read a paper that took data from upstate, rural New York, on bureaucratic actions and extrapolated it to a national level.
- Confirmation bias may prompt the researcher to select data sets and samples that match their preconceptions and ignore those that are not congruent. In theory, using a null hypothesis can mitigate this problem. However, unless the researcher is careful and understands the bias, the bias could influence the null hypothesis development and lower its mitigation effect.
- Correlation does not mean causation. Just because two concepts have a statistical correlation does not mean one causes the other. While most doctoral programs teach this concept, it is far easier to observe it than to do it. Biases, flawed data selection, and limited understanding of the observed system can influence a researcher to accept causation where it does not apply. I have seen this in many papers. It is also a fundamental issue with many social justice studies. These studies try to show race causes most social ills when that may not be the case.
- Human systems may differ from other systems. This issue is like the differences between classical and quantum physics, as discussed in the uncertainty principle. Adding humans to the system is akin to moving from classical physics to quantum physics. Humans can be unpredictable and conflicting or changing motives. See The Tragedy of the Commons: Rational Actors and Why do we act in Other than our Self-interest: Is the Rational Actor/Choice Theory Valid. For example, a lever’s effects are not dependent on the lever’s motivations. As variables change, we can calculate with near certainty to effects of the change. But peoples’ motivations and concepts of self-interest may change as people change group affiliations, social memes change, and puppet masters push agendas. For example, when I was in grade school, we respected the founding fathers of the US Republic and learned about them. Now, many schools either do not teach them at all or denigrate them. This change affects motivation and culture. Cultural changes affect actions and results, unlike the lever.
Humanity is an interesting dynamic. We are the quantum mechanics in most systems. With large numbers, we may approach classical system performance, but most sample sizes may not have enough data or the “right” data to mitigate the human quantum effects.