PUBLIC HEALTH RESEARCH METHODS
Year : 2012 | Volume
: 56 | Issue : 1 | Page : 4--11
Qualitative research in applied situations: Strategies to ensure rigor and validity
N Nakkeeran1, Sanjay P Zodpey2,
1 Associate Professor, Indian Institute of Public Health, Gandhinagar, Gujarat, India
2 Director, Public Health Education, Public Health Foundation of India, New Delhi, India
Associate Professor, Indian Institute of Public Health, Gandhinagar, Gujarat
Traditionally, qualitative studies are founded on interpretative and constructive epistemology. The process of data collection in these studies is longer and intensive. This helps to build a strong rapport with the community, hence enabling to capture the field as naturally as possible. These characteristics provide an ample scope to take care of quality and validity of data. However, in applied situations, data collection is often a truncated activity. This robs away a number of taken-for-granted strengths of traditional qualitative research methods: No time is spent on rapport building; holism is left behind, instead we engage in selection; we focus narrowly on specific phenomenon of concern, divorced from its context; analysis does not evolve out of an iterative process. In this paper, we aim to discuss some of the issues related to rigor and quality of such studies and strategies available to address them.
|How to cite this article:|
Nakkeeran N, Zodpey SP. Qualitative research in applied situations: Strategies to ensure rigor and validity.Indian J Public Health 2012;56:4-11
|How to cite this URL:|
Nakkeeran N, Zodpey SP. Qualitative research in applied situations: Strategies to ensure rigor and validity. Indian J Public Health [serial online] 2012 [cited 2020 Feb 19 ];56:4-11
Available from: http://www.ijph.in/text.asp?2012/56/1/4/96949
In a study, based on a "systematic sample of 100 trials, published in English from the register of the Cochrane Effective Practice and Organisation of Care Review Group", Simon Lewin and others find that 30 of them used qualitative methods along with the randomized control trials (RCT). From these 30 studies, they identified at least 11 "ways in which qualitative methods can be used alongside randomized controlled trials", thus highlighting the important role qualitative methods could play even in RCTs, considered as the gold standard in bio-medical research. At the same time, they also point out that the "rigor of qualitative studies undertaken alongside (these) randomized controlled trials … is an important concern", and they identify "major shortcomings in many of these studies", including "lack of a clear justification for the qualitative approach used, … inadequate descriptions of context, sampling, data collection, and analysis methods; little reflection on the researcher's role in the research process … and insufficient evidence to support the claims made in the paper". 
How do we address these issues of rigor and quality in these situations? It is encouraging to see an increasing use of qualitative methods in applied situations. A majority of applied qualitative studies in medicine or public health gets published in journals, meant for studies done with quantitative principles. Publishing in these journals not only requires a certain structure of writing but also a certain form of representing the reality and adducing evidence. There is also a trend in the direction of systematic reviews of qualitative studies. These aspirations introduce a number of restrictions on these studies in terms of design of the study as well as the structure of their publication. These restrictions are often compounded with limitations, stemming from limitation in funding and inappropriate use of computer based software for analysis. Equally, we see extensive use of qualitative methods in program implementation and evaluation by field based organizations. Here too, there are problems of lack of trained qualitative researchers at the grass-roots, anecdotal evidences masquerading as qualitative research findings. In this paper, we aim to discuss some of the issues related to rigor or quality of applied studies and elaborate on strategies available in literature to address them.
The paper aims to problematize the dichotomy between traditional long term qualitative (read ethnographic) studies and one-short applied qualitative studies and highlights the strengths and possibility of using data collection strategies and analysis that are located somewhere between these two poles. It argues that by having a systematic study design with a degree of flexibility, a staggered data collection process, an initial set of codes that is allowed to progressively evolve and an early starting of analysis, could help to retain a certain degree of inductive nature of the study and concurrent aspect of analysis and take the applied studies closer to long-term ones.
Traditional long-term vs. applied qualitative studies
Qualitative studies in anthropological research are traditionally founded on interpretative and constructive epistemology.  To suit this form of enquiry, these studies are usually intensive in nature, studying a small population. The process of data collection in these studies is usually longer with a prolonged contact with the field; it is often not a one-time interaction between the researcher and informants, but is protracted with opportunities for several meetings, from several angles and at different locations. This helps to build a strong rapport with the community that is participating in the study. The researcher is able to increasingly merge with the field and become unobtrusive. This facilitates capturing the field as naturally as possible. Qualitative research gives importance to the context in which an action takes place. Meaning of any action or phenomenon and the intentions of a behavior are derived from the context in which it occurs. A phenomenon is not broken into narrow variables plucked away from its context, but is seen as embedded on it. The researcher gets to know the participants as real people, and becomes part of their lives. S/he is in a continuous process of reflexive, iterative engagement with the field and data throughout the study. 
These unique characteristics of traditional qualitative research provide ample scope to take care of quality and validity of data. In fact, the term validity, defined in a strict sense - measuring what an instrument or method should measure - is a misnomer as far as classical anthropological methods are concerned. As Janice M. Morse argues "good qualitative inquiry must be verified reflexively in each step of the analysis… qualitative inquiry, properly conducted, is self-correcting and rigorous, and the results are strong.  Anthropological methods capture a domain intensively, richly, holistically and from many different facets unlike in a survey. Classically, qualitative studies aim to represent reality rather than finding out the truth.  "(W)e are concerned not with measurement but with description and meaning; hence, reliability and validity take on a different role".  Lincoln and Guba instead, prescribe concepts such as credibility, transferability, dependability and conformability of study findings as more relevant measures in the context of in-depth qualitative research. 
Qualitative methods are increasingly being used in applied situations. By applied studies, we refer to those studies that use knowledge, practice and theory from an academic discipline towards specific practical purposes, such as improving policy decisions or resolving a social problem rather than using merely for advancement of methods and theories of that discipline. Providing a comprehensive definition or an exhaustive typology of applied qualitative research in public health is not possible here for the sheer range in variety of these studies. However, it may be mentioned that applied studies in qualitative research could take the form of studies that aim to understand specific behavior, behavior change or barriers to change; meaning of and reasons behind such behaviors; understanding the meaning of concepts such as pain, illness, patient satisfaction, quality of care, autonomy etc. in a specific group; capturing perceptions of a population on specific products or interventions; process documentation or evaluation of interventions; situational analysis; etc.
With constrains imposed by purpose, nature of expected outcome, funding and deadlines, these studies often are characterized by one or more of the following features:
They have to be done relatively in short durations with limited scope for rapport building or understanding the context.Although methods of data collection could be the same as that are used in traditional anthropological research - such as unstructured interview, observation or case study, they have to be often used as one-short, stand-alone basis and not embedded on a prolonged fieldwork.The value of holism has to be left behind; instead, one is compelled to engage in selection.The focus has to be usually on a specific phenomenon of concern, divorced from its broader context.They often use qualitative methods for data collection, but may not necessarily use the tenets of qualitative methodology. It may be pertinent here to dwell a little deeper into this distinction. 'Qualitative methodology' denotes to a particular orientation and study designs, which subscribe to the idea that reality is socially constructed or interpreted as against trying to explain one form of reality from positivist standpoint. Qualitative methods on the other hand, refers to types of data collection methods or tools, which provide scope for collecting textual, descriptive data having no compelling relationship with epistemological orientation or methodology.  Most often, the process of analysis cannot be an iterative concurrent process, but is a truncated and often a terminal activity.
In other words, the non in-depth nature of applied studies robs away a number of taken-for-granted strengths of a traditional qualitative study. As a result, the high degree of 'internal validity', which a traditional qualitative study would guarantee, can no longer be assumed to be taken care. Instead, there are possibilities of one arriving at conclusions and policy recommendations that may not necessarily flow fully from the research but are anecdotal, impressionistic or journalistic. Hence, there is a need for use of planned strategies to ensure a higher degree of quality and rigor of the study.
In sections that follow, we aim to bring out some of the strategies available in literature that may facilitate in ensuring a higher degree of rigor and quality in applied qualitative studies. These include: (a) having a systematic study design, including plan for data collection and analysis, but with a necessary degree of flexibility, (b) employing focused activities for rapport building and getting to know the context, (c) having staggered data collection stage, (d) care in selection of informants, events, data domains, (e) systematic data archiving, (f) triangulation and member checking, (g) beginning the process of analysis early in the study and keeping it as concurrent as possible.
Before moving on, it may be useful to briefly look at a typology of qualitative data and it's implication on validity and accuracy. Janice Morse classifies qualitative data in to 3 types. a) Indirect data, which are inferential and abstract, which by definition may or may not be valid. Questions of accuracy are of little importance here. b) Semi-direct data include participants' reports of what happened. They are approximations, variation can occur between perceptions of different participants; yet there is an aspiration towards validity of representation. c) Direct data is expected to represent actual phenomena very closely. Such accuracy is emphasized in conversational analysis and micro-analytic observational studies.  It may be useful on the part of a researcher to reflect on what type of data one is dealing with.
Having a-priori study design
In an in-depth study conceptualization, research questions and the design emerge during the process of study, facilitating an inductive process. In applied situations such an inductive mode of study may not be possible. Under this situation, it is important to engage in an explicit stage of conceptualization, development of a framework and systematization of data collection plan. It is also possible that with well-defined research questions, categories for coding could also be delineated at the outset, but with scope for modifications. Nicholas Mays and Catherine Pope observe "the basic strategy to ensure rigor, and thus quality in qualitative research is systematic, self-conscious research design, data collection, interpretation and communication". 
Manuals for qualitative research are available in a number of content areas in public health, such as child health, nutrition, HIV/AIDS, reproductive health etc., which provide guidelines to design and implement studies. Although they may require further customization for specific studies and geographical areas, they definitely can provide useful frameworks to design applied studies. 
Rapport building and understanding context
In conventional qualitative studies, a researcher usually trained in research methods, immerses in to a field for relatively long period. Hence, s/he is familiar with the field area, its context and problems. Applied situations may not permit such dedicated area-specific researchers to devote very long period for getting familiar with the context. In such situations, combining some form of participatory exercise could be very useful for gaining acceptance in the community. It could be a sort of ice-breaking participatory exercise, mapping, drawing seasonal calendars, historical timelines, transect maps etc. In-depth interviews with important members of the community will prove extremely useful to understand the social, cultural and political context of the community. In case of studies connected to interventions, exhaustive review of existing documents and interviews of 'experts' outside the community could be useful to understand program inputs and structure of implementation. Community acceptance gained and insights into community structures will provide a tangible basis for selection of informants or 'cases'.
Staggered data collection
Typically, in applied studies, data collection is done as a one-time activity, spread over a short period of time. Unless a study is part of a battery of studies, it often involves a given number of key-informant interviews or/and focus group discussions (FGD), done in a rapid pace. If done in an underdeveloped region, it may also involve the research team traversing a vast geographical area in a difficult terrain, which effectively makes the available time further short. Logistically, this arrangement may be convenient, economical or sometimes the only option available.
Instead, if resources and study design permit, it may be a useful idea to stagger and spread the process of data collection over a longer period - as much as the study could permit. After the initial few interviews or FGDs, a short break will give time to take a look at the data, check the quality of data, engage in a nascent stage of analysis, accordingly revise the data collection strategy / tool and resume with another set of interviews of FGDs, and so on. This provides an opportunity for an iterative and inductive process in a limited sense. However, such staggering may not be of significant use in studies that require a uniform data collection process and a rigid framework for analysis decided at the outset of the study.
In situations, where the study is of a short duration, invariably there is a stronger role of selection rather than an emphasis on holism. A good understanding of the community structures and process is a pre-requisite for selection. In addition to in-depth interviews of important members of community or community level personnel, one could also use specific participatory tools, which bring out community structures. These could be in the form of mapping, Venn-diagrams, or card sorting which brings out political and social dynamics; it could also be flow diagrams, seasonality charts or timelines to capture processes. These participatory procedures are done in group settings thus enabling collection of rich background information on the community in relatively short time.
Traditional qualitative studies often do not make any reference to sampling. Sampling is considered to be incongruent with the spirit of qualitative methods. However, with increased use of qualitative methods in applied disciplines, this issue has been problematized in current literature.  The phrase 'theoretical sampling' has been used in grounded theory, referring to a process of reflexive selection of 'cases' "in order to develop emergent themes, to assess the adequacy, relevance, and meaningfulness of themes, to refine ideas, and to identify conceptual boundaries".  Here too, the concern is not so much on sample size as much it is on the criteria of selection. Drawing from different sources, Nancy Leech, however, goes beyond mere selection to a prescription for sample size considerations for different types of qualitative studies.  It is important to emphasize the need for a reflexive consideration while selecting the 'cases', both in terms of criteria and in numbers. However, any prescriptions on sample size will be practically inappropriate and will amount to ignoring the heterogeneity of qualitative methods and methodologies. A sampling grid could be a useful tool to enlist informants or observational units and think through the process of sampling.  This could also help in engaging in an explicit step of retrospection on sampling during later stage of analysis.
Care in selection of domains of data collection and study participants
Keeping the understanding gained on the community structures and processes as the backdrop, it would be important to make sure that the relevant and cardinal cultural domains are selected for data collection. Domains could vary across dimensions such as stakeholders, different social groups in the community, time brackets, geographical area, and departments in an organisation or even stages of a process.
Similarly, selection of study participants could be based on our understanding of community structures. Typically, informants should be able to theoretically or conceptually represent different elements of relevant context, with which the research is concerned.
Assessing the salience or centrality of observations/responses
It may be pertinent to assess the importance of observations made in terms of their salience or centrality as a contributory element of our explanation / inference. This could be done by checking how frequently is the particular observation repeated in the social milieu - is it a rare occurrence or a pre-dominant, normative occurrence. One could explore if a particular event is a dominant one such that it determines or configures other aspects of social life of the society or organisation. One could also explore if the observed event is a surface process, indicating something deeper and important aspect of the social life or organizational culture.  Looking out for negative or non-normative responses too is important as they could bring out the range of variations in the aspects studied.
Assessing validity of response
In studies using participant observation wherein we combine informal interviews with observation, the responses which are observed could be considered to be more valid than those responses which are (orally) reported. Similarly, observations in natural settings are more valid than observations made in non-natural setting; and spontaneous oral responses are more valid than responses which are elicited. Responses elicited for undirected questions are more valid than responses elicited for a leading question. Close-ended questions in most situations are leading in nature. This exercise may involve use of quasi-statistics on transcripts. It could be a useful scale to interpret and infer from a collection of field data wherein contradictory responses have been elicited from different participants or from same participant, at different point in time. In such situations, the context in which such responses were elicited assumes great significance. 
It refers to a stage in data collection when a researcher feels that no new themes, ideas, or insights are emerging and continuing the data collection does not enrich or expand explanations already arrived. It marks the end of data collection. However, in practice, it may be difficult to recognize the arrival of saturation point. Many studies refer to saturation in a non-problematic way; but it may be more fruitful to explicitly problematize and articulate how and why one thinks that saturation point has arrived, substantiated with clear evidence. 
Systematic data archiving
A qualitative researcher is expected to maintain notes in a number of formats, which capture interviews and observations made in the field. These could be field notes, methodological notes, analytical notes etc. As much as possible, these should be complete, systematically arranged or even indexed. It is a form of audit trail for the researcher if s/he wants to go back to re-asses if the inferences and interpretations made are conceptually connected and derived from the field observations. Use of instruments like checklists and field-log could also add strength to systematize the process of data collection.
As a part of field notes, it may be useful to maintain a form of reflective notes wherein the researcher documents how her/his personal position, value framework, ideological biases, etc., could have affected the way the study went about. This is especially important to understand problems in informant selection and interpretation. 
This is another common strategy, used to ensure the quality of data collected and validity of inferences. Triangulation means using more than one method to collect data on the same phenomena. Information thus collected are compared, inconsistencies are noted and taken with caution. Similarly, data on the same phenomena is collected from different vantage points in terms of location of individuals, seasons, etc. This helps to capture the variations in the aspect over the entire spatial and temporal spectrums and add to comprehensiveness of the observation-set.  Applied studies with one time data collection process often provides limited scope for only triangulation.
Member checking or respondent validation
A researcher could validate his/her account of a phenomenon or findings with the members of the community being studied. It is possible that the account of community members may vary from that of researcher.  Any significant difference between the account of researcher and that of community members could be considered as new original data and incorporated in to the explanation. This strategy, however, assumes that researchers have opportunities to interact with study participants at more than one point in time: During data collection and after preparation of preliminary report. This strategy is built-in if data collection is iterative in nature.
In a long term study, data analysis is a concurrent process with an iterative relationship with data collection. In applied studies, data analysis is most often and to a great extent a terminal activity. Problem with this situation is that, it allows no or very less scope for data analysis to feed back into data collection process. Typically, such studies will have very little openness or flexibility in design. In studies, which aim to capture the relationship between a limited set of pre-identified variables, this may not pose serious problems. The idea of 'framework analysis', suggested by Ritchie and Spencer, serves as an appropriate strategy in such situations.  Strength of framework analysis is that it allows themes to emerge from an a priori framework as well as from the field data. However, if the purpose of a study is to explore and to bring out new variables or themes, such a closed design and terminal analysis may be a serious limiting factor. Data may throw up new leads, but the study design may not provide time and scope to follow the new leads.
Staggered data collection could provide a mid path between a long term studies and one-short applied studies. Accordingly, if the objectives of the study demand and if resources permit, it would be more appropriate to design the process of analysis such a way that one is not totally kept away from the stream of data almost till the end of the data collection phase. Having a systematic plan for data analysis from the outset and initiating the process of analysis very early in the study may help to address these problems to a great extent. One could start with a set of codes and sub-codes, drawn from objectives and conceptual framework of the study and be open for adding new codes, rearranging and recoding them, and identifying new relationship between codes as a continuous iterative process.
Studies using qualitative method without proper understanding of its epistemological basis could face a number of problems which may weaken the rigor of the study.
One of the most serious problems that can threaten the quality of data, comes from a very basic problem of researcher, imposing his/her worldview on the respondent or assuming / interpreting responses from the point of view of the researcher instead of that of informant's.  This is where an understanding of the epistemology behind qualitative methods assumes the greatest significance. The purpose of an in-depth interview is to uncover the informants' frawework of meaning, opinions, interpretations and prioriteis. They may vary significantly or subtly from that of the researcher or from the 'established', 'mainstream', 'scientific' or 'modern' worldviews. It is extremely imporant to be aware of these differences and make efforts not to impose his/her own views on informants. This could happen at any stage of research, from the stge of preparing the study design or data collection to analysis and inference making. In applied research in the fields of public health and medical research, where there is a dominat medical paradigm perceived to be the right perspective, there is a greater danger of falling in to such traps.
Tendency to quantify
A most common pitfall is a tendency towards quantification. There are situations where one may successfully use quantification in a qualitative study. In a mixed-method situation, quantitative data could provide a strong basis for interpreting qualitative data. This is, however, distinct from applying principles of quantification on data collected using qualitative methods. Any attempt at quantification should be consistent with the epistemological basis of method used, sampling employed and the format of data that is being collected. It may be useful to keep in mind that 'samples' selected for interviewing or observation in a qualitative study has been not selected based on statistical sampling and the sample do not represent the universe statistically. Here, they represent the study area based on knowledge they have and information they can share on the study area. Hence, it will be a totally inappropriate to quantify responses of a collection of informants to extrapolate and generalize the data to a larger population in terms of numbers or proportions.
Abstraction using a quantitative indicator
Another common practice observed in mixed-method situations is to summarize and reduce enormous volume of qualitative data, based on a limited number of quantitative data, collected through other methods. This problem may creep in if the qualitative analysis plan is weak. Without a proper plan for analysis and data representation, a researcher may take recourse to representing the final inferences using an insufficient quantitative indicator.
Treating response in group situation as individual responses
Yet another closely related problem is the tendency to quantify responses collected in group situations. Individual responses in group situations are normative in nature and do not represent views of individuals. Hence, it may be misleading to treat them as individual responses and quantify as responses from so many individuals.
The strength of qualitative research is its interpretative nature, intensive and long duration of the study, rapport that is built with the community studied and importance given to context. In applied sitiuations, all these strengths are given up for want of time. Search for insider's meaning is given up for a search for evidence and truth. Using qualitaive methods under such circumstances has the danger of the study becoming anecdotal and journalisitc. In forgoing pages, we have discussed some of the strategeis that could be employed to ensure rigor in data collection and analysis. These strategies essentially involve compensating for shortage of time with a more rigorously prepared systematic reserch design with a degree of flexibility, relatively staggered data collection stage, an initial set of codes that progressively evolves and an early starting of analysis process. Instead of a prolonged stage of rapport building, a shorter period is spent using participatory exercises, and through efficiently involving gate-keepers and information-rich informants. It is imperative to involve in a delibrate and carefull process of selection. Clear and systematic preparation of data recording and data reduction plans are also required to enable an audit trail. Rigor of a study can also be maintained by avoiding generalizing the findings of the study beyond the warrented limits and resisting tendancy to quantify the findings derived through qualitative principles. The fundamental requirement, however, is on the part of the researcher; to be reflexive and explcit about each and every stage and component of research rather than assuming them to be unproblamatic.
|1||Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: Methodological study. BMJ 2009;339:b3496. |
|2||Nakkeeran N. Knowledge, truth and social reality: An introductory note on qualitative research, Indian J Community Med 2010;35:379-81.|
|3||Nakkeeran N. Qualitative Research Methodology: Epistemological Foundation and Research Procedures. Indian J Soc Work 2006;67:104-18.|
|4||Morse JM. Insight, inference, evidence, and verification: Creating a legitimate discipline. Int J Qual Methods 2006;5:1-7.|
|5||Mays N, Pope C. Assesing quality in qualitative research. BMJ 2000;320:50-2.|
|6||Lincoln Y, Guba E. Naturalistic Enquiry. Beverly Hills, CA: Sage; 1985.|
|7||Green J, Thotrogood N. Qualitative methods for health research. London: Sage; 2004.|
|8||Winch PJ, Wagman JA, Malouin RA, Mehl GL. Qualitative research for improved health programs: A guide to manuals for qualitative and participatory research on child health, nutrition, and reproductive health, Baltimore: Johns Hopkins University; 2000. Available from: http://sara.aed.org/publications/cross_cutting/qualitative/qualitative.pdf [Last accessed on 2010 Dec 08].|
|9||Leech NL. The role of sampling in qualitative research. Academic Exchange Quarterly 2005. Available from: http://www.thefreelibrary.com/The+role+of+sampling+in+ qualitative+research.-a0138703704 [Last accessed on 2009 Nov 16].|
|10||Charmaz K. Grounded theory: Objectivist and constructivist methods. In Denzin NK, Lincoln YS, (eds). Handbook of qualitative research. 2 nd ed. Thousand Oaks CA: Sage; 2000. p. 509-35.|
|11||Becker HS, Geer B. Participant observation: The analysis of qualitative data. In: Burgess RG, editor. Field research: A sourcebook and field manual. London: George Allen and Unwin; 1992. p. 239-50.|
|12||Bowen GA. Naturalistic inquiry and the saturation concept: A research note. Qual Res 2008;8:137-52.|
|13||Emerson RE, Fretz RI, Shaw LL. Writing ethnographic fieldnotes. Chicago: University of Chicago Press; 1995.|
|14||Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess RG, editors. Analysing Qualitative Data. London: Routledge; 1994. p. 173-94.|
|15||Britten N. Qualitative research: Quality interviews in medical research. BMJ 1995;311:251-3.|