Digital Math Assessments May Hinder, Not Help, Problem-Solving Skills

Why It Matters

Designers creating digital educational tools must be aware that the medium itself can influence user performance and the validity of the assessment. Simply digitizing existing paper-based tests without careful consideration of interface design and interaction can lead to inaccurate evaluations of student capabilities.

Key Finding

Moving math tests to computers can change how hard they are and what they measure, and some students find it a less natural way to do math, potentially adding difficulty beyond the math itself.

Key Findings

• Digitizing mathematics assessments can change task difficulty and the skills being assessed.
• Assessing 'process skills' and 'problem-solving' in mathematics via computer presents unique design challenges.
• Some students, particularly at the GCSE level, expressed skepticism towards computer-based examinations.
• The computer may not be a 'natural medium' for mathematics, potentially adding an extra cognitive load.

Research Evidence

Aim: To investigate the challenges and implications of designing computer-based assessments for mathematics, particularly concerning the measurement of process skills and problem-solving, and to compare these with traditional paper-based methods.

Method: Comparative analysis and design research exercise

Procedure: The study involved analyzing existing mathematics assessment tasks, comparing paper and computer versions of a commercial test, and undertaking a design research exercise to create, implement, and trial a computer-based assessment system for similar tasks. This included analyzing equating studies and gathering student feedback.

Context: Mathematics education (primary and secondary)

Design Principle

The digital medium's affordances and constraints must be intentionally designed to align with the learning objectives and assessment goals, rather than simply replicating existing analog formats.

How to Apply

Before developing a digital assessment, conduct user testing with the target audience to identify potential usability issues and ensure the interface does not interfere with the assessment of core mathematical skills.

Limitations

The study's findings regarding student skepticism might be context-specific and could evolve with increased digital literacy and familiarity with computer-based testing.

Student Guide (IB Design Technology)

Simple Explanation: Making math tests on computers can be tricky. Sometimes, the computer itself makes the test harder or measures the wrong things, not just the math skills. It's important to design computer tests carefully so they actually test what they're supposed to.

Why This Matters: This research highlights that the design of the interface and the choice of digital medium can significantly impact the effectiveness and validity of an assessment, a crucial consideration for any design project involving educational technology.

Critical Thinking: To what extent does the 'naturalness' of a medium for a particular task influence the validity of assessments conducted within that medium?

IA-Ready Paragraph: The transition of traditional assessment formats to digital platforms requires careful design consideration, as evidenced by research suggesting that computer-based mathematics assessments can inadvertently alter task difficulty and the skills being measured. This highlights the need for a user-centered design approach that accounts for the unique affordances and potential cognitive load introduced by the digital medium, ensuring that the technology serves to enhance, rather than impede, the accurate assessment of student capabilities.

Project Tips

• When designing a digital tool for a specific subject, consider how the interface might affect the user's ability to perform the core task.
• If comparing digital and physical prototypes, ensure the core functionality is equivalent, but acknowledge that the medium itself is a variable.

How to Use in IA

• Use this research to justify the importance of user testing and iterative design when developing digital assessment tools, emphasizing how the medium can influence outcomes.

Examiner Tips

• Demonstrate an understanding that the 'digital' aspect of a solution is not neutral and can introduce its own design challenges and effects on user performance.

Independent Variable: Format of assessment (computer-based vs. paper-based)

Dependent Variable: Task difficulty, skills assessed (process, problem-solving), student performance, student perception/attitude

Controlled Variables: Mathematical content of the tasks, student age/educational level (within specific comparisons)

Strengths

• Direct comparison of paper and computer versions of a commercial test.
• Inclusion of a design research exercise to build and trial a new system.

Critical Questions

• How can digital interfaces be designed to actively support and assess higher-order thinking skills like problem-solving, rather than just presenting information?
• What are the long-term implications for mathematics education if digital tools become the primary means of assessment?

Extended Essay Application

• Investigate the impact of different user interface designs on the performance of students attempting complex mathematical problems in a digital environment.

Source

On computer-based assessment of mathematics · UPT. Syiah Kuala University Library (Syiah Kuala University) · 2010