Standardizing Theory-Driven Review Methods Enhances Research Rigor and Dissemination

Why It Matters

In design practice, rigorous evaluation of existing research is crucial for informed decision-making and innovation. Standardized methods ensure that complex qualitative and mixed-method findings are accurately captured, assessed, and communicated, leading to more robust evidence bases for design projects.

Key Finding

Clear guidelines and standards are needed for theory-driven reviews to ensure their quality and usefulness, and these can be developed through expert consensus and practical application.

Key Findings

• Existing literature on realist and meta-narrative reviews lacked comprehensive methodological guidance and clear publication standards.
• An iterative process involving literature review, expert consensus, and real-time application was effective in developing robust standards and training materials.
• Dissemination of these standards is crucial for improving the quality and consistency of such reviews.

Research Evidence

Aim: To develop and refine methodological guidance, publication standards, and training materials for realist and meta-narrative reviews to improve their quality and utility.

Method: Mixed-methods approach combining literature review, expert consensus (Delphi method), and prospective refinement on real-world reviews.

Procedure: The project involved collating existing literature, analyzing published reviews to identify gaps, using an online Delphi method with experts to establish standards, developing training materials, and prospectively testing and refining these outputs on ongoing reviews.

Sample Size: Delphi panel comprised 37 members for realist reviews and 33 for meta-narrative reviews.

Context: Health services research, but applicable to any field requiring synthesis of complex qualitative or mixed-method evidence.

Design Principle

Standardization of complex research synthesis methods enhances rigor and applicability.

How to Apply

When undertaking a review of existing literature for a design project, consider using or adapting established methodological frameworks to ensure a systematic and rigorous approach.

Limitations

The Delphi method relies on expert opinion, which can be subject to bias. The specific context of health services research may influence the generalizability of the developed standards to other domains without adaptation.

Student Guide (IB Design Technology)

Simple Explanation: Making rules for how to do certain types of research reviews makes the results more trustworthy and easier for others to use.

Why This Matters: Understanding how to systematically review complex information helps you build a strong foundation for your design project by ensuring you're basing your work on the best available evidence.

Critical Thinking: To what extent can standardized review methodologies fully capture the nuances of qualitative and mixed-method research, and what are the potential trade-offs between standardization and flexibility?

IA-Ready Paragraph: The RAMESES project highlights the critical need for standardized methodological guidance and publication standards in complex research syntheses, such as realist and meta-narrative reviews. By developing and refining these standards through expert consensus and practical application, the project aimed to enhance the rigor, transparency, and utility of such reviews, ultimately contributing to a more robust evidence base for decision-making in relevant fields.

Project Tips

• When reviewing literature for your design project, look for established methodologies that ensure a thorough and unbiased assessment of existing work.
• Consider how you can clearly document your review process to demonstrate its rigor and validity.

How to Use in IA

• Reference the development of standardized review methodologies as a justification for your chosen literature review approach.
• Discuss how adhering to established standards enhances the credibility and reliability of your research findings.

Examiner Tips

• Demonstrate an understanding of the importance of methodological rigor in research synthesis.
• Show how you have applied established standards or justified deviations from them in your literature review.

Independent Variable: Availability of methodological guidance and publication standards.

Dependent Variable: Quality and rigor of realist and meta-narrative reviews.

Controlled Variables: Expert panel composition, Delphi round structure, training material content.

Strengths

• Involved a multidisciplinary expert panel.
• Used a combination of literature review, expert consensus, and prospective testing.

Critical Questions

• How can these standards be adapted for use in fields outside of health services research?
• What are the long-term implications of adopting these standards for the diffusion of complex review methodologies?

Extended Essay Application

• Investigate the development and application of standardized methodologies for systematic reviews in a specific design-related field.
• Analyze how the rigor of evidence synthesis impacts the innovation process within a particular industry.

Source

Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project · Health Services and Delivery Research · 2014 · 10.3310/hsdr02300