Conceptual Frameworks, Incommensurability, and the Replication Crisis in Psychology

Conceptual Frameworks, Incommensurability, and the Replication Crisis in Psychology

I tweaked someone else's image using GIMP. You can find the original at

Recently I had to justify my continued existence as a Masters student by writing a proposal for external review as well as giving a succinct ten minute presentation on my topic to an open forum. Both were a success and I have now had my candidature officially confirmed. As a result of my feedback, my focus is beginning to reorient a little. Primarily, I will be narrowing down so as to ensure I can present a decent argument in only 30,000 words. So while what I'm about to write go into below only gives a broad suggestion to what I'm researching, nevertheless I thought it might be useful to adapt into a post.

My research thesis will contend that an obscured incommensurability between and within rival psychological definitions is one of the causes behind the replication crisis in contemporary psychological science. In more common language, the central idea of my research is investigating if one of the multitude of reasons why psychological researchers sometimes struggle to replicate the findings of other researchers, is that the different individuals are unaware they are incorrectly assuming a shared definition of all concepts involved between the two studies. From this position, I suggest that to be able to talk about psychology through the lens of concepts such as incommensurability it is important to first address the discourse around whether it is valid to consider psychology a science.

Because it encapsulates such idiosyncratic issues as mental health and consciousness, in my research I see that psychology is sometimes argued as not belonging to the category of science. However, to me, to argue if psychology is or is not a science, requires a firm grasp on the characteristics of science. Toward this, I’ve become quite attached to the simplistic definition of science provided by Alan Chalmers in his book What is This Thing Called Science? To quote Chalmers, “take argument and the available evidence seriously and do not aim for a kind of knowledge or a level of confirmation that is beyond the reach of available methods." One way to explain this is that any knowledge that logically aligns with observation and analysis can be thought of as science. But, it is also important to remember that what is accepted as knowledge, observation, and analysis are all embedded in the cultural milieu of the time. The first part of my project is to establish if this is a reasonable way to categorise science.

Once I have a working definition for the characteristics of modern science, I will apply this analysis to psychology. So far, my intuition is that one of the tensions that exists when people don’t want to label psychology as a science, lies in the blurred edges between clinical psychology and experimental psychology. I intend to explore if it is reasonable to demarcate between clinical and experimental psychology when considering psychology as a science. If it is reasonable for psychology to be categorised as a science, then I will investigate the way in which the umbrella term of the replication crisis is being used to critique psychological research.

For at least a decade now (possibly significantly longer), one of the major concerns facing all fields of science has been the failure of experimental reproducibility. Some commentators immediately assume that failure to reproduce the findings of a given piece of research suggests the researchers involved were either incompetent or dishonest. However, while unfortunately this has been the case with some research, I disagree that it is true of all, or even the majority. In my research I will explore the arguments that suggest the causes of replication failure are as varied as the scientific fields wherein, they are observed. So far, my research suggests that the identified cause of a replication failure in a specific physics experiment, may have little in common with the identified cause of failure within a genetics experiment, yet both are categorised under the umbrella replication crisis.

I tweaked someone else's image using GIMP. You can find the original at psychology, the causes of replication failure are just as varied as in other sciences. However, due to an identified historical culture of questionable research practices, I feel the discussion invalidly becomes hyper-focused on statistical manipulation or intentional fraud. While questionable research practices are being increasingly discouraged, and a higher level of research standards are being encouraged, there is still more work that needs to be done regarding simple communication of methodological practices between the original experimenters and those attempting to replicate their research.

The crunch of my thesis is in exploring if I am correct that sometimes researchers fail to replicate others results, simply because they are embedded in different conceptual paradigms, thus interpreting the intended meaning of certain phrases differently than the original researcher.

Under the umbrella of philosophy, much has been written on the imperfect nature of language for conveying meaning. Building upon the discourse around both Gottlob Frege and Ludwig Wittgenstein’s writing on the philosophy of language, I aim to explore how philosophers of science have applied their ideas to scientific discourse. In particular, I see parallels between Frege’s sense and reference, Wittgenstein’s language-games, and the concept of incommensurability. From here, I will explore some of the ideas concerning how disharmonious meaning might inadvertently contribute to replication failure.

David Chalmers has argued that sometimes when two thinkers are at disagreement the problem lies in the idiosyncratic meaning each is attributing to a contentious phrase. If both parties were to take the time to unpack the specific concept, Chalmer’s suggests that their disagreement may naturally dissolve. Again, I am exploring how this might relate to science through the lens of incommensurability.

In Thomas Kuhn’s writings on scientific paradigms through the ages, he argued that it is sometimes difficult, if not impossible, to compare between disparate scientific frameworks because the meaning behind key words and phrases is so distinct as to be incomparable, or as it argued incommensurable. While I am currently more familiar with the concept of incommensurability as it has evolved out of the works of Kuhn, I am also exploring how the concept has developed through the writings of other thinkers such as Paul Feyerabend, and Ludwick Fleck. It may also be that there is some overlap between incommensurability and the feminist philosophy use of the concept ‘situated knowledge’ as it applies to the philosophy of science.

It is this train of thought that lead me to explore if failure to identify this disconnection, what I am calling obscured incommensurability, is an observable factor contributing to the replication crisis in psychology.

Despite not exactly agreeing on the details, thinkers like Stanley Klein, David Trafimow, and Brian Earp have proposed that replication failures within psychological research lie in either poorly defined primary or auxiliary theories and assumptions. I am exploring the validity of their arguments while also considering the arguments proposed by Daniel Kahneman that suggests the issues often lies in problems with vague methodologies. I will explore if the arguments the four thinkers I just mentioned are making fit within the scope of the philosophical arguments for conceptual frameworks and incommensurability.

If I am successful in explaining the identified issues as a form of incommensurability between the concepts and frameworks different research teams are leveraging, then I will propose that this understanding can be developed into a post-hoc diagnostic tool to be used when a replication failure occurs. I propose that it is unreasonable to expect every researcher to have complete conceptual parity with every other researcher. Even though desirable, it may still be unreasonable to expect this on even a smaller scale wherein one research team is attempting to replicate the results of another. Rather, I propose that when replication fails it should be standard practice that researchers perform a post-hoc analysis of both studies to ensure both where comparable. One of the factors I am proposing researchers should be considering is if there was any previously obscured incommensurability between theories, auxiliary assumptions, and or methodologies.

I tweaked someone else's image using GIMP. You can find the original at this as my aim, I anticipate three major objections to my suggestion that obscured incommensurability is attributing to the replication crisis in psychology.

Firstly, as I have already mentioned there is some debate about whether psychology should be classified under the umbrella of science. Therefore, as incommensurability is a philosophy of science concept it may be invalid to explore psychology through the lens of conceptual frameworks and incommensurability. However, as I previously mentioned, I suspect that one of the easiest ways to establish psychology as a science is to clearly demarcate between the clinical application of psychology as being non-scientific, while leaving the evidence-based exploration of psychological concepts under the umbrella of science. I hold some concern that readers may interpret my classification as clinical psychology as non-scientific to imply a negative view of the field. This would be incorrect. I will endeavour to clearly define the application of science as being distinct to the practice of science. As such, wherein clinical psychology is the application of psychological research, it is distinct from the practice of psychological science.

Secondly, it may be suggested that I am overstating incommensurability because science uses clearly defined language and standards. However, this argument appears to be historically inaccurate. I will compare this position against philosophical and historical arguments around meaning variation in the progression of broader scientific understanding. Combining this with the definition of science I ultimate settle upon, I will consider if it is reasonable to conclude that ‘mature science’ is ever entirely homogeneous.

Lastly, even if my intuition of obscured incommensurability is found to be valid, it may be suggested that it is a statistically insignificant problem. This is a concern I share. However, while it is within the scope of this project to test the validity of my thesis, if true it would require a secondary project of larger scale to effectively analyse the statistical occurrence of obscured incommensurability. It is possible that I have already identified the topic of my PhD.

In closing, to summarise the objectives of my research, first I aim to find a reasoned definition of the characteristics of science and explore how it might be applied to the field of psychology. Assuming I conclude it appropriate to consider experimental psychology as a science, I will evaluate the discourse around the replication crisis in psychology. From this position I will consider if it is valid to suggest that one of the causes of replication failure is the result of obscured incommensurability between rival conceptual meanings. I will then propose how this problem may be addressed by future researchers.