NASP Communiqué, Vol. 34, #5
February 2006
Research Reviews
The Role of Qualitative Research Methods in Evidence-Based Practice
By Adena B. Meyers & Brent A. Sylvester
Assistant Ed. Note: In this issue (and occasionally in future issues),
the Research Committee deviates from its typical two-column format of research
reviews and outcome evaluation protocols to focus on a specific research
paradigm—Steve Landau
With the increasing emphasis on evidence-based practice in school psychology
and other mental health disciplines, there is a tendency to equate the idea
of “evidence” with quantitative data produced in the context
of experimental or well-controlled quasi-experimental research. Indeed, the
majority of published evidence conforms to these standards, and provides invaluable
information about the effectiveness of various strategies and techniques. Nevertheless,
a growing number of researchers have recently begun to point out the important
complementary role that qualitative research methods can play in the development,
implementation, and evaluation of evidence-based practice in the field of school
psychology. In fact, last spring, the Journal of School Psychology published
a special issue devoted to this topic (Nastasi & Schensul, 2005a).
Although our knowledge base about effective school-based interventions is
increasing, the practitioner still must contend with a number of questions
when attempting to address the problems of specific children in any particular
local setting. Qualitative research methods may be especially useful in answering
questions that are relevant to practice but are difficult to address using positivistic (i.e.,
experimental and/or quantitative) designs. For example: What adaptations to
an existing social skills curriculum will allow it to be implemented successfully
in a special education classroom in which the students have a particular set
of special needs? How feasible will it be to introduce an empirically supported
school-wide discipline system into a building, given the existing norms related
to discipline and system change? How can home-school collaboration be improved,
given the needs and constraints of the families in a particular community?
Why do teachers follow-through with some, but not all, of the interventions
generated in consultation with a particular school psychologist?
Although qualitative methods can be enormously helpful in answering questions
of this type, the terminology and methods of qualitative inquiry may be relatively
unfamiliar to the traditionally trained school psychologist. Thus, we begin
with a discussion of definitions and key concepts pertinent to qualitative
research. This is followed by an illustration of some of the potential contributions
of qualitative research to the field of school psychology, in which we describe
examples of published qualitative studies. Finally, we argue that qualitative
researchers rely upon many of the same skills that practicing school psychologists
already possess and routinely use.
Overview and Definitions
Qualitative research methods represent a vast array of practices that have
been described extensively elsewhere (e.g., Denzin & Lincoln, 1994; Glaser & Strauss,
1967; Lincoln & Guba, 1985; Miles & Huberman, 1994; Schensul & LeCompte,
1999; Strauss & Corbin, 1990). Merriam (1998) defines qualitative research
as “an umbrella concept covering several forms of inquiry that help us
understand and explain the meaning of social phenomena with as little disruption
of the natural setting as possible” (p. 5). She describes several types
of qualitative methods including ethnography (a set of strategies
developed by anthropologists for studying cultures and cultural phenomena), phenomenology (an
approach to research that emphasizes subjective experience and seeks to uncover
the essence of that experience), grounded theory (an inductive approach
to inquiry in which theory emerges from observations of specific “real-world” phenomena),
and case study (in-depth examination of a single unit). This list
is far from exhaustive. Other qualitative approaches include participatory
action research (Lewin, 1951), discourse analysis (Potter, 2003),
and naturalistic inquiry (Lincoln & Guba, 1985); and the list
goes on.
Shared Characteristics
Merriam (1998) argues that the various approaches to qualitative research
tend to share the following characteristics:
Qualitative methods focus on understanding constructed
meaning. Whereas positivistic research assumes that phenomena
are best understood from an objective standpoint, qualitative research
assumes that meaning and knowledge are constructed in a social context.
Qualitative researchers seek to understand research participants’ subjective
perspectives. The distinction that anthropologists make between the emic and etic perspectives
is relevant here. Etic refers to an outside perspective that may
represent the view of a cultural outsider or a researcher seeking to understand
a phenomenon that is separate and apart from him or herself. The emic perspective
is an inside perspective – that of a cultural insider or in many
instances the clients or research participants whom the social scientist
seeks to understand. Although researchers generally occupy an etic position,
the qualitative researcher actively seeks to incorporate the emic view
into the inquiry. Often this is accomplished by including research participants
as members of the research team (e.g., see Fine, Torre, Boudin, Boen, Clark,
et al., 2003).
The qualitative researcher is viewed as the primary instrument
for data collection and analysis. Merriam explains, “Data
are mediated through this human instrument, the researcher, rather than
through some inanimate inventory, questionnaire, or computer” (p.7).
Qualitative methods depend upon the researcher’s ability to process
information, respond sensitively to social cues, and adjust the research
design as data are interpreted and new questions emerge. The subjectivity
of the researcher is thus viewed as a resource to be leveraged rather than
a source of unwanted bias and invalidity that must be minimized.
Qualitative research tends to emphasize inductive reasoning.
Positivistic research uses a deductive approach to inquiry. The researcher
begins with general or abstract ideas (theory), which form the basis of specific
predictions (hypotheses), which are tested in highly specific, logically contrived
situations (experiments). Theories are developed and refined through this hypothetico-deductive
process. Qualitative research in contrast seeks to build theory from observations
of real-world phenomena in their specific and natural contexts. Although the
researcher often begins with some a priori theoretical ideas, the
goal is not to test these ideas against the data, but rather to develop a theory
that makes sense of data generated in the context of a particular investigation.
d) Fieldwork is used as a primary mode of data collection.
Qualitative researchers seek to study phenomena in context. Rather than sending
out surveys in the mail or bringing participants into a laboratory, qualitative
researchers usually conduct observations and interviews in the environment
of interest. Qualitative researchers must spend sufficient (sometimes extensive)
amounts of time in the field to ensure that they thoroughly understand the
phenomenon in context, and that the conclusions being drawn are credible. This
process is sometimes referred to as prolonged engagement.
e) Whereas positivistic research attempts to present precise findings using
quantitative strategies to summarize data, qualitative studies
aim to provide rich descriptions of phenomena. Although some
data reduction does occur in qualitative research, the goal is not to reduce
the data as much as it is to present them holistically and comprehensively,
within a credible theoretical framework.
Coding and Analyzing Qualitative Data
Qualitative data are usually obtained through interviews, observations, and
document review (Nastasi & Schensul, 2005b). Rigorous qualitative studies
typically use coding procedures that are systematic, recursive, and collaborative.
Webster-Stratton and Spitzer (1996) describe a three-level coding method based
on Glaser and Strauss’s (1967) Grounded Theory. Level I coding
(also known as open coding) involves breaking the data down into small,
meaningful units, often based on the informants’ own words. During level
II coding (also known as axial coding), codes are grouped into categories.
The categories are developed, revised, and validated through a method known
as constant comparison. Level III coding or selective coding involves
exploring relationships among core categories. Since the process is recursive,
researchers do not necessarily move through these levels in a linear manner,
but rather move back and forth between levels as needed. For example, Level
II or III coding might raise questions that require revision of some Level
I codes, or even collection of additional data.
Establishing Scientific Rigor in Qualitative Designs
In positivistic research, the quality of a study is judged on the basis of
such factors as the validity and reliability of the measures and the generalizability
of the data. In qualitative research the rigor of the study is equally important
but different procedures are used to assess it. Lincoln and Guba (1985) provide
definitions of five key concepts that can be used to evaluate the scientific
integrity of a qualitative study: (a) trustworthiness, (b) credibility,
(c) transferability, (d) confirmability, and(e) dependability.
Trustworthiness is a general term that refers to
the overall reliability and validity of qualitative research. Each of the other
four concepts can be construed as dimensions of trustworthiness.
Credibility refers to the appropriateness and accuracy
of the data sources and interpretations. For example, were the proper informants
identified and interviewed? Were their responses complete and truthful? If
a study of home-school collaboration concluded that parent involvement in education
was low due to lack of parent time and interest, but the only informants were
a handful of highly involved parents, the findings would lack credibility.
Furthermore, if parents were interviewed in the presence of the principal,
the credibility of their responses would be questionable, as parents might
not feel free to provide complete and accurate information in that particular
context. Often, qualitative researchers attempt to improve the credibility
of their data by conducting multiple interviews with the same informants. This
may increase the completeness and accuracy of the findings because of enhanced
trust between the researcher and participants, and because follow-up questions
can be developed to clarify ambiguities and fill in gaps from earlier interviews.
Many well-designed qualitative studies also use a procedure known as member
checking that involves soliciting feedback from participants about the
emerging coding system and interpretations of the data. According to Lincoln
and Guba (1985), “The member check, whereby data, analytic categories,
interpretations, and conclusions are tested with members of those stakeholding
groups from whom the data were originally collected, is the most crucial technique
for establishing credibility” (p. 314).
Transferability is related to the idea of representativeness
and is concerned with the contextual boundaries of the findings. In general,
qualitative research does not place a high priority on generalizability in
the abstract, because the aim is usually to answer questions that are relevant
in particular, local contexts. When a qualitative researcher thoroughly understands
the context under investigation, and provides a rich, contextualized, description
of the phenomenon of interest, the reader is better able to make inferences
about the transferability of the findings.
Confirmability is similar to the idea of replicability,
but as Webster-Stratton and Spitzer (1996) point out, “…qualitative
research almost by definition defies the criterion of reproducibility.” The
variables that may influence the outcome of the study are usually difficult
to control, and often are integrally part of the context. Controlling or manipulating
these factors would defeat the purpose of the investigation. Furthermore, since
the researcher is the primary instrument in qualitative research, it is expected
that each investigation will be idiosyncratic, influenced by the researcher’s
subjective engagement with the material being studied. Nevertheless, rigorous
qualitative studies make the research process transparent. Researchers keep
detailed records (sometimes referred to as an audit trail) that would allow
for an independent review of the data collection, coding, and analysis procedures.
The auditor would not be able to replicate the study, but would be able to
review in detail what the researchers did, why they did it, and how they arrived
at their conclusions.
Dependability refers to the reliability of the
coding procedures. Positivistic researchers establish reliability by calculating
reliability statistics based on percentage of agreement between raters, internal
consistency of scales, and so on. Since qualitative research usually involves
a collaborative coding process with multiple raters, some qualitative researchers
choose to calculate inter-rater reliability in the traditional way. However,
as noted above, coding and analysis of qualitative data usually involve a recursive
process, so a single measure of inter-rater reliability would often be insufficient
for establishing dependability of the coding procedures. Codes and codebooks
may be refined and revised several times within one investigation. Qualitative
research teams often meet repeatedly to discuss, compare, and revise codes.
According to Lincoln and Guba (1985), researchers can establish dependability
through their audit trail. Similar to a fiscal auditor, an “inquiry auditor” would
examine the record-keeping procedures as well as the products of the investigation
(e.g., findings and interpretations). Dependability depends on the former,
whereas confirmability depends on the latter.
These concepts represent a sample of the many procedures that qualitative
researchers use to establish scientific rigor. Recently, the Interdisciplinary
Qualitative Research Subcommittee (IQRS) of the Task Force on Evidence-based
Interventions in School Psychology (sponsored by the Society for the Study
of School Psychology and the American Psychological Association’s Division
of School Psychology) developed detailed guidelines for evaluating qualitative
research. The interested reader is encouraged to consult Nastasi and Schensul
(2005b) for more information about these guidelines.
Potential Contributions of Qualitative Methods to Evidence-Based Practice
in School Psychology
In the past decade, a great deal of attention has been devoted to establishing
empirical support for various school-based interventions to improve children’s
academic, behavioral, social, and emotional functioning. However, even if an
intervention has been shown to work in several experimental demonstrations,
there is no guarantee that it can be successfully transported to a new setting
with different clients, different treatment providers, and different contextual
circumstances. Qualitative research can help fill these gaps by answering questions
about social validity, transportability, and cultural factors that traditional
efficacy studies are often unable to fully address. Below we elaborate on each
of these issues and offer illustrative examples of relevant qualitative research.
Social validity from the participants’ perspective. An
efficacious intervention is unlikely to work in a new context if the providers
or recipients consider its goals and outcomes to be unimportant or its procedures
inappropriate. A number of researchers interested in these issues have investigated
the constructs of social validity and treatment acceptability.
According to Lyst, Gabriel, O’Shaughnessy, Meyers, and Meyers (2005),
most of this research has involved participants filling out rating scales after
reading vignettes or viewing videotapes about hypothetical intervention situations.
They argue that social validity can be understood more fully through qualitative
research methods aimed at elucidating participants’ subjective perspectives
about the interventions they are actually experiencing.
To that end, Lyst et al. (2005) conducted a mixed (qualitative and quantitative)
method study in which they interviewed parents and teachers participating in
an early intervention program for kindergarteners identified as being at risk
for school failure. They related the themes uncovered from the interviews to
participants’ scores on an existing measure of treatment acceptability.
Their findings were generally consistent with prior theory, but also provided
important information about parent and teacher perceptions of the intervention
that were not revealed in the rating scale. For example, both the interviews
and the rating scale scores indicated that teachers perceived the intervention
to be only minimally disruptive, but the interview data also provided details
about some of the specific disruptions that teachers did experience (e.g.,
problems with scheduling), as well as information about effective strategies
the service providers used to minimize disruption (e.g., arriving to the classroom
early for scheduled intervention sessions so that they could touch base during
the teacher’s planning time).
Transportability. When school psychologists implement
interventions in naturalistic settings, they must often do so without the benefit
of the intensive training and on-going feedback that is typically provided
during intervention research. According to Siegal (2005), although research
has shown that cooperative learning is an effective method of instruction (when
researchers closely monitor the teachers delivering it), less is known about
teachers’ efforts to implement cooperative learning techniques on their
own. To explore this question, she conducted a two-year ethnographic study
that included multiple classroom observations and interviews with five teachers.
She found that the teachers differed in their understanding and implementation
of the concepts and techniques of cooperative instruction. Of note, she observed
that although all teachers implemented most of the cooperative learning techniques,
some teachers systematically omitted techniques related to facilitating students’ social
skills and group process. Further, she found that teachers’ experience
level and conceptual understanding of cooperative learning interacted with
implementation.
Culture specificity. Although the field of school
psychology is concerned with issues of cultural diversity, intervention research
rarely considers cultural variables. For researchers and school psychologists
interested in delivering services in multiple or diverse cultural contexts,
this poses a significant problem. Given their emphasis on studying phenomena
in context and their roots in anthropology, qualitative research methods can
be especially helpful in addressing cultural issues relevant to school-based
interventions. According to Nastasi (1998), “Culture specificity implies
that critical elements of the intervention (e.g., intervention strategies and
targeted competencies) are relevant to the targeted culture, make use of the
language of the population, and reflect the values and beliefs of the members
of the culture (p. 169).” Through their work with the Sri Lanka Mental
Health Project, Nastasi and her colleagues have used a combination of ethnographic
and participatory action research methods to develop, implement, and evaluate
culture specific school-based mental health services in Sri Lanka (e.g., Hitchcock,
Nastasi, Dai, Newman, Jayasena, et.al., 2005; Nastasi, Varjas, Bernstein, & Jayasena,
2000; Varjas, Nastasi, Moore, & Jayasena, 2005).
Qualitative Research and the School Psychologist
The above examples illustrate the potential role of qualitative research
methods in the design, implementation, and evaluation of evidence-based practice.
Although the idea of engaging in qualitative research as part of school psychology
practice may seem intimidating, it should be noted that the skills needed to
conduct qualitative research are similar to the clinical skills that school
psychologists routinely employ in the assessment process. In fact, Fagan and
Wise’s (2000) description of school psychologists’ clinical assessment
skills overlaps considerably with the skills that Merriam (1998) attributes
to effective qualitative researchers. The relevant skill sets can be grouped
into the five broad areas of: (a) developing and evaluating hypotheses, (b)
reviewing documents, (c) interviewing, (d) behavioral observations, and (e)
personal characteristics.
Developing and evaluating hypotheses. Both the skilled
clinician and the qualitative researcher must be able to collect and evaluate
data relevant to the questions under investigation. To accomplish these, competent
investigators and clinicians usually conduct interviews with parents, teachers,
and children (or other key informants or research participants). They also
systematically review records and documents and conduct behavioral observations
in the relevant contexts (such as classrooms or other school settings). In
their interpretation of the data, school psychologists and qualitative researchers
look for correspondence between different types and sources of data (i.e.,
multiple methods, informants, settings).
Reviewing documents. While conducting assessments,
school psychologists may examine medical records or school records that document
academic performance, attendance, discipline referrals, permanent products,
or results of previous testing. Qualitative researchers may review organizational
records (e.g., meeting minutes, policies), public records (e.g., memos, newsletters),
and their own private records (e.g., journals, field notes). School psychologists
generally review records to obtain information on baseline performance and/or
changes over time. Similarly, qualitative researchers often review records
to obtain information on conditions prior to the study and/or changes over
time.
Interviewing. Before conducting an interview, researchers
and practitioners are ethically obligated to provide the interviewee with information
about the interview and how the data will be used, and they explicitly obtain
informed consent. According to both Fagan and Wise (2000) and Merriam (1998),
skilled interviewers build rapport by asking non-threatening questions early
in the interview, using the language of the interviewee, and following the
pace of the interviewee. They also avoid closed-ended or “yes/no” questions,
multiple questions combined into one, coercive questions, and leading questions.
Fagan and Wise recommend that clinicians compare perspectives of different
interviewees, whereas Merriam recommends that qualitative researchers synthesize
data across informants.
Behavioral observation. When conducting
behavioral observations, qualitative researchers and school psychologists strive
to analyze the relationship between behavior and its consequences. They describe
the context of the target behaviors (setting, participants, activities), and
they can increase the validity or trustworthiness of their findings by increasing
the number and duration of observations. The major difference between school
psychologists and qualitative researchers conducting observations is that the
school psychologist generally is not a participant in the setting being observed,
whereas the qualitative researcher may be functioning as a participant observer.
Personal characteristics. Fagan and Wise (2000)
note that ideal personal characteristics of skilled clinicians include flexibility,
resistance to stress, and interpersonal warmth. Similarly, Merriam (1998) posits
that skilled researchers possess tolerance for ambiguity and empathy. These
authors also suggest that skillful school psychologists and qualitative researchers
have strong communication skills (both written and verbal) and show high levels
of interpersonal sensitivity.
Conclusion
Because of the potential for qualitative research to contribute to evidence-based
practice, qualitative research methods represent a useful set of techniques
for the scientist-practitioner. Although the terms and concepts associated
with qualitative research may be unfamiliar to traditionally trained school
psychologists, many of the processes that qualitative researchers use are actually
quite similar to professional activities in which many school psychologists
engage on a routine basis. As a result, many elements of qualitative research
may readily be incorporated into the scientific practice of school psychology.
References
Denzin, N.K., & Lincoln, Y.S. (Eds.). (1994). Handbook of qualitative
research. Thousand Oaks, CA: Sage.
Fagan, T.F. & Wise, P.S. (2000). School psychology, past, present,
and future. Bethesda, MD: NASP.
Fine, M., Torre, M.E., Boudin, K., Boen, I., Clark, J., Hylton, D., Martinez,
M., Missy, Roberts, R.A., Smart, P., & Upegui, D. (2003). Participatory
action research: From within and beyond prison bars. In P.M. Camic, J.E. Rhodes, & L.
Yardley (Eds.), Qualitative research in psychology: Expanding perspectives
in methodology and design. (pp. 173-198). Washington, DC: American
Psychological Association.
Glaser, B.G. & Strauss, A.L. (1967). The discovery of grounded theory.
Chicago: Aldine.
Hitchcock, J.H., Nastasi, B.K., Dai, D.Y., Newman, J., Jayasena, A., Bernstein-Moore,
R., Sarkar, S., & Varjas, K. (2005). Illustrating a mixed-method approach
for validating culturally specific constructs. Journal of School Psychology,
43, 259-278.
Lewin, K. (1951). Field theory in social science: Selected theoretical
papers. New York: Harper.
Lincoln, Y.S. & Guba, E.G. (1985). Naturalistic Inquiry. Thousand
Oaks, CA: Sage.
Lyst, A.M., Gabriel, S., O’Shaughnessy, T.O., Meyers, J., & Meyers,
B. (2005). Social validity: Perceptions of check and connect with early literacy
support. Journal of School Psychology, 43, 197-218.
Merriam, S.B. (1998). Qualitative research and case study applications
in education. San Francisco: Jossey-Bass.
Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis: An
expanded sourcebook. (2nd ed.). Thousand Oaks, CA: Sage.
Nastasi, B.K. (1998). A model for mental health programming for schools and
communities. School Psychology Review, 27, 165-174.
Nastasi, B.K. & Schensul, S.L. (Eds.). (2005a). Contributions of qualitative
research to the validity of intervention research [Special Issue] Journal
of School Psychology, 43(3).
Nastasi, B.K. & Schensul, S.L. (2005b). Contributions of qualitative
research to the validity of intervention research. Journal of School Psychology,
43, 177-195.
Nastasi, B.K., Varjas, K., Bernstein, R., & Jayasena, A. (2000). Conducting
participatory culture-specific consultation: A global perspective on multicultural
consultation. School Psychology Review, 29, 401-413.
Potter, J. (2003). Dicsourse analysis and discursive psychology. In P.M.
Camic, J.E. Rhodes, & L. Yardley (Eds.), Qualitative research in psychology:
Expanding perspectives in methodology and design. (pp. 73-94). Washington,
DC: American Psychological Association.
Schensul, J.J. & LeCompte, M.D. (1999). Ethnographer’s toolkit. Walnut
Creek, CA: Altamira Press.
Siegal, C. (2005). An ethnographic inquiry of cooperative learning implementation. Journal
of School Psychology, 43, pp. 219-239.
Strauss, A.L. & Corbin, J. (1990). Basics of qualitative research:
Grounded theory procedures and techniques. Thousand Oaks, CA:
Sage.
Varjas, K. Nastasi, B.K., Moore, R.B., & Jayaena, A. (2005). Using ethnographic
methods for development of culture-specific interventions. Journal of School
Psychology, 43, pp. 241-258.
Webster-Stratton, C. & Spitzer, A. (1996). Parenting a young child with
conduct problems: New insights using qualitative methods. Advances in Clinical
Child Psychology, 18, 1-62.
© 2006, National Association of School Psychologists. Adena
B. Meyers, PhD, is an Associate Professor in the Department of Psychology
at Illinois State University. Her research focuses on family, community,
and school-based interventions, adolescent pregnancy and parenthood, and
child maltreatment. Brent A. Sylvester, BS, is a doctoral student
in School Psychology in the Department of Psychology at Illinois State University.
His research focuses on ecological influences on infant development, including
infant-caregiver attachment and maternal self-efficacy.