American College Of Surgeons - Inspiring Quality: Highest Standards, Better Outcomes

Reviewing Qualitative Research in Surgical Education Literature

Robert D. Sinyard, MD, MBARoi Anteby, MDCarl Gustaf S. Axelsson, MD, MPhil, MMScMichael G. Healy, EdDHalle B. Ellison, MD, FACS


Medical education research has increased since its recognition as a distinct field of study.1 The field is diverse and includes quantitative, qualitative, and mixed-methods methodologies. This practical guide is a resource for surgeons to strengthen their understanding and evaluation of qualitative research by focusing on study design and methodology, data collection and analysis, and limitations. It may be used as a resource for reading and interpreting qualitative research and providing peer review feedback.

Qualitative Methodologies

Qualitative research differs from quantitative research in many important ways. The most profound difference is the epistemology—or view on reality—that each technique is founded upon. Quantitative research is based upon objectivism or positivism, the belief in the existence of an absolute truth that, through effective research, can be discovered or described.2 Alternatively, qualitative research is rooted in the philosophy of constructivism, the view that reality is a construct of historical, social, and individual perspectives. Therefore, the focus is interpreting or understanding social settings and/or human experiences.3 Research questions that seek to understand, explore, describe, or explain experiences are suited for qualitative research. Examples include questions starting with "how," "what," or "why," which are not fully answered using numbers and statistical analyses.3 Commonly used qualitative methodological approaches include case study, ethnography, grounded theory, and phenomenology (Table 1).

Table 1: Commonly Used Qualitative Methodological Approaches

Case Study

Seeks to understand a case or case-series that reflects an underlying problem of interest4

Ethnography Seeks to understand people in their cultural and social setting4
Grounded Theory Seeks to understand social processes in order to inform development of new theory4,5
Phenomenology Seeks to understand the meaning of individuals' lived experiences6

The goal of data collection in qualitative research is depth of understanding, which places particular importance on subject selection and engagement.7 Subjects can be selected through a variety of sampling methods (Table 2).

Table 2: Common Sampling Methods2

Convenience Sampling Subjects are selected based on availability and accessibility to the research team
Purposive Sampling Subjects are selected based on pertinent experiences or perspectives
Random Sampling Subjects are selected completely by chance from the general population or a subset of interest
Snowball Sampling Selection of future subjects is based upon the recommendations of current subjects

Though differences exist in each methodological approach, similar techniques are used for data collection. Common techniques include content analysis, interviews, and direct observation of social interactions. Unlike power analyses that dictate sample size in quantitative research, the concept of saturation drives the data acquisition process in qualitative research. Saturation is achieved when the data collection process no longer uncovers novel insights or themes.6

Qualitative Data Analysis

Qualitative manuscripts should clearly describe the data analysis process. There are many ways to investigate a question. The quality of data analysis does not depend on a single approach or interpretation. Each of the approaches can be equally rigorous. The process typically begins with a systematic review, or "coding," of data. Coding represents the link between data and the investigator's interpretation of those data. Coding may be done inductively or deductively.5 An explanation should be provided for why an approach was used, based on the research question and methodology. Components of the analysis process for interview data may be described as follows:

Initial codes emerged while reviewing a sampling of data focused on the research question. Using the initial codes, a codebook was developed, which was iteratively refined as further data was reviewed. Similar codes were grouped to form categories, which were used to structure the codebook.

Qualitative software can help support and organize the coding process but using software is not required. Once data are coded, investigators look for themes emerging from the categories. A simplified example of the analysis process is illustrated in Figure 1.

Figure 1. Example of thematic analysis process for qualitative interview data

Figure 1. Example of thematic analysis process for qualitative interview data

An analysis may include a variable number of themes. Traditionally, two to five themes are included. The significance of a theme relates to the ability to address the research question, not how often it appears. For studies with multiple coders, the Kappa coefficient may be used to determine intercoder reliability or the consistency among coders to apply the same codes to the same text.8 Investigators may use additional techniques to promote quality in the analysis process (Table 3).

Table 3. Examples of Quality Techniques

Audit Trail Maintaining detailed research records, including changes occurring during the research process9
Member Checking Inviting participant feedback on data and/or interpretation of data10
Triangulation Combining different perspectives, methods, and data sources to verify convergence on specific themes4

Presentation of Data and Discussion

Study results should be presented sequentially and include detailed explanations of the themes identified, paired with data, such as illustrative participant quotes. Data may be presented either within the text, as a table, or in a figure, such as a concept map. Data selected to highlight themes should be logical and transparent to both investigators and readers. Based on the presentation of themes, investigators may create a model or framework, which can illustrate the results and how results add and relate to current literature.

Specific considerations apply to the discussion of qualitative results. The human and social experiences studied in qualitative research are complex. The research product is not expected or required to describe every facet of the phenomenon under investigation. Results should reflect the concept of dependability in that they are consistent with the context in which they were created.10 Results should also reflect credibility in the sense that they are trustworthy and believable. Investigators should further discuss the transferability to other settings of the facets they do address. This concept aligns with the concept of generalizability in quantitative research.10 As qualitative data are obtained by the investigator, the investigator's background and experience can influence how they interpret, process, and code data, including coding differently than co-investigators, or from participants' intent. This influence should be clearly described, and the confirmability of the findings should be based on whether the results reflect participant responses as opposed to investigator biases.10 Critical readers will interpret the results based on their perception of the rigor of the research (Table 4).4

Table 4. Aspects of Rigor4

Reflexivity Has the investigator's perspective about and influence on the work been considered?
Adequacy Was the data sample adequate to represent the breadth, depth, and nature of the phenomenon being studied?
Authenticity Is the data an authentic reflection of the phenomenon being studied?
Trustworthiness Are analysis techniques described clearly and carried out systematically?
Resonance Are interpretations of data meaningful for those who live in the social experience studied?

Tools for Evaluating Qualitative Research

Formal and informal checklists exist to guide study design and writing and can assist in the peer review of qualitative research. These checklists are limited in application and do not guarantee quality research. However, they can be used as reference tools for constructing and/or evaluating the methodology and reporting of qualitative studies. The two most utilized and cited tools are the Standards for Reporting Qualitative Research (SRQR), and the Consolidated Criteria for Reporting Qualitative Research (COREQ).11,12 Both require a fundamental understanding of the components of qualitative research and may be used when designing or reviewing qualitative studies.


Qualitative research is an important component of medical education that can expand knowledge and deepen understanding of complex phenomena. Surgical educators should develop skills to understand, interpret, and review qualitative education research.

Summary of Learning Points

  • Qualitative research enriches understanding of complex phenomena in education and requires a different analytic approach than quantitative studies for both readers and reviewers.
  • Evaluation of qualitative education literature should include consideration of the research question, study design, methodology, data collection, analysis techniques, and presentation of data.
  • Reference tools may aid in evaluating the rigor of qualitative education literature. 


The authors would like to thank Dr. Roy Phitayakorn for facilitating collaboration on this article.

About the Authors

Robert D. Sinyard, MD, MBA, is a general surgery resident currently completing a surgery education research and simulation fellowship at Massachusetts General Hospital, Harvard Medical School in Boston, MA. 

Roi Anteby, MD, is a postdoctoral surgery education research and simulation fellow at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Carl Gustaf S. Axelsson, MD, MPhil, MMSc, is a postdoctoral medical education researcher at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Michael G. Healy, EdD, is a research fellow at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Halle B. Ellison, MD, FACS, is an assistant professor of surgery and palliative care at Geisinger Commonwealth School of Medicine, Danville, PA.