In the early 2010s, psychology researchers highlighted a high prevalence of questionable research practices1, exposed several notable cases of academic fraud2, and revealed alarmingly low reproducibility rates for results from prominent studies3. These revelations led to ongoing attempts to improve psychological research, with a focus on how research should be transparent and accessible to everyone. These principles are at the heart of ‘open research’, a movement dedicated to improving the openness of scientific scholarship. UNESCO prioritised open research explicitly in its 2021 global agenda4, demonstrating growing international momentum towards open research practices across disciplines.

Although progress has been promising, the open research conversation has been dominated by a narrow quantitative methodological tradition. Many concerns about questionable research practices in psychology exist only in quantitative research, such as P hacking (manipulating analytical choices to maximize the chances of obtaining a statistically significant result), hypothesizing after results are known (HARKing) and data fishing (searching through data to find statistically significant results in the absence of a pre-determined hypothesis)1. Thus, many of the tools proposed to combat such practices — including pre-registration and making data and code publicly available — also derive from a view of rigour that is rooted in experimental, quantitative and positivist assumptions. That is, these tools promote a view that valuable knowledge is derived only from empirical, observable evidence. Such tools therefore promote objective measurement, replicability and statistical generalizability.

These standards are not relevant to qualitative research, and problems occur when they are uncritically transferred without carefully considering the epistemological underpinnings. Likewise, treating use of these tools as a blanket indicator of the quality or rigour of work is also problematic. This quantitatively based view of rigour marginalizes the value that qualitative researchers bring to research and to conversations about open research.

Incompatible standards

Qualitative research operates under fundamentally different philosophical assumptions from quantitative research. As opposed to quantitative positivism, qualitative research typically draws from interpretivism and constructivism, which focus on understanding meaning, experience and context. Such approaches also emphasize the role of the researcher as an active participant in the knowledge-building process. Thus, there is a tension and inequity when qualitative scholars are increasingly encouraged to engage with open research practices that are based on the assumptions of quantitative research.

For instance, open data sharing is increasingly incentivised (and in some cases mandated) by journals and funders as a mechanism to enhance the reproducibility of research findings and to enable readers to retrace the analytical steps taken by researchers. This approach is valid and valuable in the context of quantitative research, where reproducibility is often an appropriate benchmark. However, for certain qualitative methodologies and analytical approaches, reproducibility is not a compatible standard. It is well-documented that researcher positionality (a researcher’s own worldview and stance towards their research), context, culture and research ontology all shape how a researcher makes analytical decisions about their data. This concept is the crux of reflexivity, which refers to the active and intentional consideration of how the researcher shapes their research5. Reflexivity is relevant to all research paradigms but is a hallmark of rigorous and thoughtful qualitative work. Thus, reproducibility, or the ability for other researchers to exactly retrace the steps of another, is not analytically possible or appropriate for some approaches6. A lack of data sharing in qualitative research does not indicate poor quality, and a ‘lack of reproducibility’ here is not a flaw; it is a feature of qualitative approaches. If the goal of a study is to understand the lived experiences of a sample (rather than to uncover patterns that can be generalized to a wider population) and the active role a researcher plays in constructing this understanding is acknowledged, then lack of reproducibility is not a problem. Thus, it is unclear what function open data has for qualitative methodologies.

There is a similar question of the epistemological compatibility of specifying an analytical plan a priori with qualitative methodologies. The practice of study pre-registration is increasingly heralded as a marker of rigour (although there is some debate about its utility7). However, it is not always appropriate for qualitative researchers to specify, with precision and detail, their analytical steps ahead of data access, because the most suitable analytical approach might depend on the breadth, richness and depth of the data, which in many cases cannot be known a priori. For example, if written responses to free text questions on a qualitative survey are short and relatively superficial, they might be best analysed used a descriptive content analysis approach. But if participants provide long and detailed responses, more interpretive analytical approaches might be more appropriate. In this instance, a failure to specify the analytic strategy in advance is an appropriate and valid — not questionable — research practice.

Current open research conversations are not flexible enough to acknowledge the nuance and context-specificity of qualitative methodologies, which differ vastly in their epistemological and ontological underpinnings8. Some approaches to qualitative analyses are positivist-leaning in their orientation, whereas others are more interpretative and constructivist. Some qualitative researchers might value reproducibility, whereas for others it is not appropriate. In some cases, open data sharing and pre-registration could benefit qualitative scholars. For example, open data sharing might be useful to facilitate secondary analyses, and pre-registration might be useful as a pedagogical training tool9. As mandates for openness and transparency increase, there is a growing risk of reinforcing the assumption that all open research tools are universally applicable across diverse research methodologies.

Learning from qualitative traditions

Open research efforts are united by principles and values that include equity, fairness, responsibility and flexibility4. Such values are not only aligned with qualitative approaches but arguably derived from qualitative traditions. Qualitative researchers are trained to extensively document their processes, to keep detailed and reflexive field notes, and to confront and critically reflect upon their biases and analytical blind spots10. They are well-versed in a slow, thoughtful, democratic and collaborative approach to scholarship. There are therefore some well-established practices within qualitative research that can meaningfully contribute to openness across other methodologies, a direction of transfer that has been notably absent from the open research conversation.

For example, qualitative researchers are particularly well-versed in considering how their personal, professional or political biases might shape and inform their research. This consideration is the crux of reflexivity, a hallmark of rigorous qualitative research. As open research advocates in psychology continue to wrestle with how, or whether, to address researcher bias, there is much to be gained from engaging with the body of reflexive work developed by qualitative methodologists5,11. Indeed, perhaps the work of quantitative scholars could be more rigorous and thoughtful if they adopt some of the reflexive practices that derive from qualitative scholarship5.

Further, there might be value in adopting other qualitative research practices and principles more broadly. For example, qualitative researchers often have a fundamentally different approach from quantitative scholars to how research participants are valued in the research process. Many qualitative approaches value and promote co-production and collaboration with research participants12, with the aim of promoting a non-extractive approach to research with more balanced power dynamics between researcher and participants. In particular, participatory methodologies advocate for the value of working with participants at all stages of the research process, to ensure that findings genuinely reflect participants’ experiences in a robust and transparent way. This approach offers a different way of thinking and rigour, by centring participant perspectives and honouring the experience that they bring to knowledge production. Researchers in the quantitative domain are largely unfamiliar with this approach. However, there might be value in all scholars adopting participatory approaches in which they work with participants in a more democratic, collaborative way towards more rigorous research.

Moving forward

It is important to consider how qualitative researchers might respond to emerging standards of rigour and how qualitative traditions can inform and strengthen other methodological approaches. Open research must be open to all to achieve a diverse and inclusive research culture. Psychology needs epistemic, methodological, and epistemological pluralism to examine phenomena from different perspectives. The approach to research reform must be flexible, collaborative and equitable13. Without these values, open research is simply a set of practices that perpetuates an increasingly narrow ideal of what rigour looks like.

I urge policymakers to take seriously the notion that open research practices are not epistemologically appropriate for all types of research. Mandating such practices inherently excludes qualitative scholars and pushes them further into the margins, which limits knowledge production. Thus, open research guidelines should be flexible, with the opportunity to genuinely opt-out where appropriate. Assumptions about what constitutes rigour should also be continuously critically inspected. Finally, although articulating the incompatibility of open research and qualitative methods is a useful step, there is also value in stressing how quantitative scholars can learn from the traditions of qualitative research. Methodological siloes harm everyone, so it is time for all researchers to engage in more cross-methodological sharing to work towards genuinely open research.