Open Science Practices Lag in Research Posters, Hindering Reproducibility and Inclusivity

Why It Matters

In design practice, the ability to reproduce results and understand the context of user research is paramount for building reliable and effective solutions. When research is not transparent or inclusive, it limits the scope of application and can lead to biased design decisions.

Key Finding

Most research posters analyzed did not share data, preregister their studies, or report on replications, and predominantly featured research from American researchers on Western populations.

Key Findings

• Research presented was heavily skewed toward quantitative studies by American researchers using Western hemisphere samples.
• Sharing of data/materials, preregistrations, and replications were uncommon.
• The findings highlight a gap between the growing appreciation for metascience issues and their practical implementation in research dissemination.

Research Evidence

Aim: To assess the prevalence of inclusive and reproducible science practices within research posters presented at a major scientific conference.

Method: Meta-scientific analysis of research artifacts.

Procedure: Researchers analyzed 2615 research posters from the 2021 SRCD biennial meeting, evaluating the extent to which they incorporated transparent research practices (e.g., data sharing, preregistration, replication studies) and inclusivity (e.g., diverse sample representation).

Sample Size: 2615 posters

Context: Academic research dissemination (psychology/child development).

Design Principle

Research underpinning design decisions should strive for transparency, reproducibility, and diverse representation to ensure broad applicability and validity.

How to Apply

When reviewing literature for a design project, critically evaluate the methodology for transparency (e.g., are methods clearly described? Is data available?) and sample diversity. Consider how these factors might limit the applicability of the findings to your target user group.

Limitations

The study focused on poster presentations, which may not fully represent all research conducted or disseminated within the field. The findings are specific to the context of the SRCD meeting in 2021.

Student Guide (IB Design Technology)

Simple Explanation: Researchers aren't sharing their data or methods openly enough, and most studies are only about people from Western countries. This makes it hard for others to check their work or use it for different groups of people.

Why This Matters: Understanding how research is conducted and shared helps you critically evaluate the information you use for your design projects, ensuring your designs are based on sound and relevant evidence.

Critical Thinking: How might the lack of open and reproducible practices in academic research directly lead to flawed or biased design solutions?

IA-Ready Paragraph: Research presented in academic settings, such as the SRCD biennial meeting, often lacks transparency and inclusivity. A meta-scientific investigation of 2615 posters revealed that sharing of data/materials, preregistrations, and replications were uncommon, and studies were heavily skewed toward American researchers using Western hemisphere samples. This limits the reproducibility and generalizability of findings, posing a challenge for designers who rely on such research to inform their practice.

Project Tips

• When citing research, note if the authors have made their data or methods available.
• Consider the diversity of the sample in the research you use – does it represent your intended users?

How to Use in IA

• Reference this study when discussing the limitations of existing research or justifying your own transparent research methods.

Examiner Tips

• Demonstrate critical evaluation of sources by acknowledging potential biases or limitations in research methodology, such as lack of transparency or diverse sampling.

Independent Variable: Research poster characteristics (e.g., inclusion of data sharing, preregistration, sample demographics).

Dependent Variable: Prevalence of transparent and inclusive research practices.

Controlled Variables: Type of research (quantitative/qualitative), researcher nationality, sample origin.

Strengths

• Large sample size of research posters analyzed.
• Focus on critical metascience issues relevant to scientific rigor.

Critical Questions

• What are the systemic reasons for the low adoption of open science practices?
• How can design fields actively promote and integrate these practices into their own research and development processes?

Extended Essay Application

• Investigate the transparency and inclusivity of research cited in a specific design field or for a particular design problem.

Source

A metascience investigation of inclusive, open, and reproducible science practices in research posters at the 2021 SRCD biennial meeting · Child Development · 2023 · 10.1111/cdev.14059