Why Evidence Reviews Are Not Research Studies

Julee Briscoe Waldrop, DNP, FNP, PNP, NC-BC, CNE, FAANP, FAAN
Staci Reynolds, PhD, RN, ACNS-BC, CPHQ, FAAN

Writer’s Camp Counselor and Guest Counselor

---

A systematic search strategy supports practice change; a systematic review answers a research question.

---

If you are conducting an evidence-based practice (EBP) or evidence-based practice quality improvement (EBPQI) initiative, you must include a review of the evidence to guide your practice change. When you write the results of your initiative for publication, you must keep the purpose of this evidence review in mind. For any EBP or EBPQI initiative, the purpose of the search of the evidence is to find, critically appraise, and synthesize the highest quality, most current evidence to support the recommendation for a practice change. While this requires a systematic search strategy, this does not require a systematic review of the literature. A systematic review is a type of research that uses explicit, reproducible methods to identify, appraise, and synthesize all relevant studies addressing a focused research question.¹

Evidence Search for Evidence-Based Practice

Evidence-based practice in nursing and health care was derived from evidence-based medicine and has broadly been received as the “right thing to do.” It was shepherded into nursing by Fineout-Overholt and Melnyk with their research and textbooks, now used almost endemically by nursing programs in the United States and globally.^2,3 The founding of the Helene Fuld Institute for EBP extended the reach of EBP from academia to practice settings. Somewhere along the way the search for the highest quality, most current evidence on which to base practice recommendations and ideally practice change (EBP) became conflated with the research method for systematic reviews. This practice often prevents authors—and you—from seeing the forest for the trees and may hinder progress, as you are burdened with essentially conducting a research study rather than simply finding the best evidence.

Evidence Search for Systematic Reviews

The purpose of the search for evidence in systematic reviews is to answer a specific research question. Research results or outcomes that fit pre-specified eligibility criteria are collated, critiqued, and synthesized. Bias is minimized by using explicit, systematic methods documented in advance with a protocol. Depending on the design and analysis of the research studies, this may include a statistical meta-analysis.⁴

Issues

Several recurring issues arise when authors conflate systematic search strategies with systematic review methodology. So much focus has been put on the systematic search strategy that this process has become the primary focus of EBP. Practice change recommendations should be based on high quality, current research evidence; however, these recommendations don’t really mean anything if practice and outcomes don’t improve.

As editors and reviewers for multiple journals, we see this problem frequently. From the purpose or the evidence searching question (such as PICO [Population, Intervention, Comparison, Outcome] or PPCO [Population, Problem, Change, Outcome]), the authors proceed to then loosely follow the process outlined in the PRISMA checklist for systematic reviews.⁵ Issues commonly seen are arbitrary search dates, such as limiting searches to articles published in the last 5 years; using limiters to define inclusion/exclusion criteria; including a document with the full search string for each database; or stating only articles available in full text are included (potentially biasing results of the search). Authors may also state a specific number of studies are reported to be included but then only a portion are cited in the literature synthesis (e.g., 15 studies were identified in the evidence search but only 8 are cited) with no explanation of why. A PRISMA flow chart—required for a systematic review research study—is also often included even though the manuscript is reporting results of an EBP or EBPQI initiative and is not a systematic review. Lastly and most problematic is that there is often no critical appraisal of the included studies.

While practice change does not always have to be based on a preponderance of high-quality evidence (depending on the risk of harm) knowing the quality of the research increases the trust in recommended practice change. Sometimes there is a mention of the level of the study based on its design (e.g. per an evidence hierarchy) but no determination of the quality of the body of the evidence that the authors are using to recommend the practice change. Critical appraisal using valid critical appraisal tools such as those developed by the Joanna Briggs Institute (JBI) or the Centre for Evidence-Based Medicine (CEBM) is often missing.^6,7

For example, in an EBP report the review of the literature (evidence) included six cross-sectional studies, two qualitative studies, and one randomized controlled trial. The authors reported that each of these studies was rated using a critical appraisal tool as moderate to high quality; however, there was no mention of the quality of this evidence as a whole. Based on the preponderance of cross-sectional studies (6 out of 9) and small number of participants, the overall quality would likely be rated much lower. This does not mean this evidence didn’t support the practice change and that benefits of change outweighed the risks, but it does reduce the rigor and clarity of this foundational part of EBP and EBPQI initiatives.

Strategies for Reporting EBP or EBPQI Initiatives

When writing a manuscript, you must provide clarity. It all begins with the purpose of the evidence review. If the purpose is to find the highest quality and most recent evidence to identify recommendations for a practice change, then start with a clear evidence searching question, either PICO or PPCO, to guide the systematic search process.^8,9 A concept table provides a structured way to translate this evidence searching question into a systematic yet flexible search strategy. A concept table related to the evidence searching question can then be used to develop the search strategy (Table 1). In this table, the PPCO question is: “For nurses working in emergency rooms (Population) with burnout (Problem) what evidence-based practices (Change) impact burnout rates (Outcome)?” When using the PPCO searching question format, the search is not biased as you do not need to choose an intervention a priori without fully reviewing the literature.

---

Table 1. Concept Table for PPCO

Concept	Focus	Keywords (Synonyms)
Concept 1	Problem	Burnout; Compassion fatigue
Concept 2	Population	Emergency room nurses; Registered nurses employed in the emergency room
Concept 3	Change	Evidence-based practices; Practices; Strategies; Measures; Policies; Protocols
Concept 4	Outcome	Burnout; Compassion fatigue

---

These keywords and synonyms can be put together with the appropriate Boolean operators and limiters for each database searched; the available evidence will be found on what improves and does not improve emergency room nurse burnout.

The critical appraisal of the quality of the studies or other evidence used to support the practice change is also necessary. A succinct synthesis of only the data/evidence related to the practice change intended outcome is required. A synthesis table supports this appraisal by allowing authors to examine the quantity, quality, and consistency of the evidence as a whole. A synthesis table can help you visualize the quantity, quality, and direction of the evidence. Table 2 clearly portrays the information needed to determine the quantity, quality and consistency of the results of the research related to the outcome of interest. The table is followed by a narrative statement which answers the questions of quality, quantity and consistency of results. The narrative follows Table 2, after the table notes (in italics).

---

Table 2. Outcome Synthesis Summary

Study	Study Design	Level of Evidence*	Critical Appraisal**	Outcome Direction
Study 1	Cross-sectional	3	Quality B	Negative
Study 2	Qualitative	6	Quality A	Negative
Study 3	Qualitative	6	Quality A	Negative
Study 4	Cross-sectional	3	Quality B	No change
Study 5	Cross-sectional	3	Quality B	Negative
Study 6	Randomized controlled trial	2	Quality B	Negative
Study 7	Cross-sectional	3	Quality B	No change
Study 8	Cross-sectional	3	Quality B	Positive

*Level of evidence based on the hierarchy selected by the authors (in this example, Level 6 = lower-level evidence and Level 1 = higher-level evidence).

**Critical appraisal conducted using the Johns Hopkins Nursing Evidence-Based Practice Critical Appraisal Tool.

Eight studies provide evidence regarding the proposed practice change (1-9).  Five were cross-sectional (1,4,5,7,8) studies, two were qualitative (2,3) and one was a randcomized controlled trial (RCT) (6). However, the majority of studies were rated moderate quality (B) (1,4-8) or high quality (A) (2,3) using the Johns Hopkins Nursing Evidence-Based Practice Critical Appraisal tool. Therefore, the overall quality is predominantly lower-level observational designs with low to moderate confidence in the evidence outcome which was largely negative with limited consistency across the studies.

---

From the synthesis table and narrative, you can answer the following questions: 

1. What quantity of evidence is there to support the recommended practice change? Are there only 1 or 2 studies that evaluated a small number of participants, or are there many studies that evaluated hundreds of participants? 
2. What is the quality of the evidence? This is where authors must assess the level of evidence and use critical appraisal tools (CAT). 
3. How consistent are the results? Quantity, quality, and consistency will determine how confident you can be in your recommendation.

The answers to these questions will form the critical appraisal. A succinct synthesis of the outcome of interest is all that is needed to help guide the practice change recommendation. As shown, this process will provide you with the most recent, highest quality evidence that will help you determine which practice change(s) to recommend and implement; an in-depth systematic review is not needed. Following these strategies will help ensure that you clearly describe your evidence searching process.

Conclusion

Distinguishing between a systematic search strategy and a systematic review allows you to apply appropriate rigor without misrepresenting the nature of your work. Clear reporting of evidence searching and appraisal strengthens EBP and EBPQI manuscripts and supports meaningful practice change—without the need for a research study when one is not warranted.

References

1. Ahn, E.J. & Kang, H. Introduction to systematic review and meta-analysis. Korean Journal of Anesthesiology. 2018;71(2):103-112. doi: 10.4097/kjae.2018.71.2.103
2. Melnyk BM, Gallagher-Ford L, Long LL, Fineout-Overholt F. The establishment of evidence-based practice competencies for practicing registered nurses and advanced practice nurses in real-world clinical settings: proficiencies to improve healthcare quality, reliability, patient outcomes, and costs. World Views on Evidence Based Nursing, 2014;1(3). https://doi.org/10.111/wvn.12021
3. Melnyk BM, Fineout-Overholt (Eds). Evidence-Based Practice in Nursing & Healthcare: A Guide to Best Practice. 5^th Edition. 2023. Wolters Kluwer.
4. Cumpston M, Flemyng E, Thomas J, Higgins JPT, Deeks JJ, Clarke MJ. Chapter I: Introduction [last updated August 2023]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane, 2024. Available from cochrane.org/handbook
5. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;n71. https://doi.org/10.1136/bmj.n71
6. Joanna Briggs Critical Appraisal Tools. https://jbi.global/critical-appraisal-tools
7. University of Oxford, Centre for Evidence-Based Medicine. Critical Appraisal Tools. https://www.cebm.ox.ac.uk/resources/ebm-tools/critical-appraisal-tools
8. Waldrop J, Dunlap JJ. CE: Beyond PICO—A new question simplifies the search for evidence. American Journal of Nursing. 2024;124(3):34-37. https://doi.org/10.1097/01.NAJ.0001007676.91191.dd
9. Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club. 1995;123(3)A12-A13.  

Authors: Julee Waldrop and Staci Reynolds

Staci Reynolds is a Clinical Professor at Duke University School of Nursing, where she teaches in the DNP program. Her clinical background includes neuroscience care and infection prevention. Her scholarship focuses on evidence-based practice and quality improvement, and she is Editor-in-Chief of the Journal of Nursing Care Quality.

Reviewed and Edited by: Leslie H. Nicoll

Copyright © 2025 by Writer’s Camp, Julee Waldrop, and Staci Reynolds. CC-BY-ND 4.0

Citation: Waldrop JB, Reynolds S. Why evidence reviews are not research studies. The Writer’s Camp Journal, 2025; 1(3):19. 10.5281/zenodo.17945800