Over-reliance on quantitative metrics can stifle genuine innovation in design research.

Why It Matters

Design practice thrives on creativity, experimentation, and the exploration of complex human needs. When assessment frameworks prioritize easily quantifiable metrics, they risk devaluing the very processes that lead to breakthrough solutions. This can discourage designers from pursuing ambitious or unconventional projects.

Key Finding

While metrics have a role, an overemphasis on them can negatively impact the quality and innovation of research by failing to capture its full value and discouraging novel approaches.

Key Findings

• Quantitative metrics can be useful for certain aspects of research evaluation.
• Over-reliance on a narrow set of metrics can lead to unintended consequences, such as gaming the system or discouraging risk-taking.
• The complexity and qualitative nature of some research, including design, are not always well-captured by standard metrics.
• A more balanced approach, incorporating qualitative assessment and expert judgment, is often necessary.

Research Evidence

Aim: To what extent does the current emphasis on quantitative metrics in research assessment hinder the exploration of novel and impactful design solutions?

Method: Expert Review and Policy Analysis

Procedure: The review involved a multidisciplinary group of experts who analyzed the role and impact of various metrics in research assessment and management, drawing on their collective experience in scientometrics, research policy, and academic administration.

Context: Research assessment and management frameworks within academic and research institutions.

Design Principle

Evaluate design research holistically, recognizing both quantitative and qualitative contributions to innovation and impact.

How to Apply

When proposing or evaluating design research projects, ensure that the assessment criteria acknowledge the value of exploration, user-centered insights, and qualitative outcomes, in addition to any measurable results.

Limitations

The review's findings are primarily focused on academic research assessment and may not directly translate to all industry design contexts.

Student Guide (IB Design Technology)

Simple Explanation: Don't just count things; understand the quality and creativity behind design work. Focusing too much on numbers can stop new and brilliant ideas from happening.

Why This Matters: Understanding how research is evaluated helps you frame your own design projects effectively and advocate for the value of your creative process and user-centered approach.

Critical Thinking: How can design research frameworks be adapted to better capture and reward the inherent creativity, user empathy, and systemic impact of design solutions, moving beyond a purely metric-driven evaluation?

IA-Ready Paragraph: The evaluation of design research necessitates a balanced approach that transcends purely quantitative metrics. As highlighted by Wilsdon et al. (2015), an over-reliance on easily measurable outputs can inadvertently stifle innovation by devaluing the exploratory nature of design and the qualitative insights gained from user-centered processes. Therefore, when assessing the impact of a design project, it is crucial to consider a wider range of criteria, including the novelty of the approach, the depth of user understanding, and the potential for long-term societal or user benefit, rather than solely focusing on easily quantifiable performance indicators.

Project Tips

• When presenting your design project, highlight the innovative aspects of your process and the qualitative user feedback you received, not just the final product's performance metrics.
• Consider how your project's success might be measured beyond simple quantitative data, especially if it involves user experience or emotional design.

How to Use in IA

• Reference this insight when discussing the limitations of quantitative data in evaluating the success or impact of your design project, particularly if your project involved novel approaches or subjective user experiences.

Examiner Tips

• Ensure your evaluation of a design project considers the breadth of its contribution, including process innovation and user-centered insights, not just easily quantifiable outcomes.

Independent Variable: Emphasis on quantitative metrics in research assessment.

Dependent Variable: Level of innovation and exploration in design research projects.

Controlled Variables: Type of research institution, funding body policies.

Strengths

• Comprehensive review by a multidisciplinary expert panel.
• Addresses a critical issue in research management and policy.

Critical Questions

• What are the specific quantitative metrics that are most problematic in design research evaluation?
• How can qualitative assessment methods be standardized and made robust enough for research evaluation?

Extended Essay Application

• An Extended Essay could explore the development of a novel, mixed-methods evaluation framework for design research projects, drawing on the principles of this review to balance quantitative and qualitative assessment.

Source

The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management · Kent Academic Repository (University of Kent) · 2015 · 10.13140/rg.2.1.4929.1363