Attention and Comprehension Checks 🎯

Understanding the Difference

Attention Check Comprehension Check
Were they paying attention? Did they understand?

Types of Checks

Attention Checks

Simple checks to verify participant engagement:

✅ Examples:

  • “For quality assurance, please click ‘6’ from the options below”
  • “What’s the day after Friday that comes before Sunday?”
  • “What did the previous question ask you to do?”

✅ Best practices:

  • Keep them simple
  • Space them throughout the survey
  • Don’t make them tricky
  • Never use force response

Comprehension Checks

Verify understanding of materials:

✅ Examples:

  • “What were the names of the couple in the vignette?”
    Options should include:
    • Sophie/Paul
    • Paul/Robin
    • Max/Robin
  • Questions testing understanding of key concepts
  • Verification of scenario details

Implementation Guidelines

Placement

  • Space checks throughout the survey
  • Don’t cluster them together
  • Consider placing after key information
  • Avoid putting them right after consent

Response Options

  • Use clear, unambiguous answers
  • Include plausible incorrect options
  • Don’t make them unnecessarily difficult
  • Consider including “I don’t know” options

🚨 Exclusion Policies

If you plan to eliminate people from the study entirely (i.e., not pay them) based on failing attention or comprehension checks , a very specific procedure must be met:

YOU MUST declare this to your participants, BEFORE the experiment begins.

  • You can do this by including a ‘Read Carefully’ question

  • The check MUST be of a specific type and conform to Prolific’s rules, as detailed here.

Note: when setting up the study in Prolific, you have to click ‘manually approve’ – rather than ‘automatically approve’– to be able to deny a submission in the first place.

These procedures do not have to be followed if you intend to pay participants who fail checks but simply intend to exclude their data from analyses.

⚠️ Critical: Please review Prolific rules and regulations regarding participant exclusion (i.e., not accepting submission) where you deny to pay the participants!! (https://researcher-help.prolific.com/en/article/3405a6)

Documentation Requirements

For Your Pre-registration

  • Number and type of checks
  • Placement in survey
  • Exclusion criteria
  • Handling of failed checks

For Your Ethics Application

  • Justify use of checks
  • Explain exclusion criteria
  • Detail payment policies
  • Describe participant notification

Quick Reference Guide

Check Type Purpose Example
Simple Attention Basic engagement “Please click ‘6’”
Logic Attention Active reading “What’s the day after Friday…”
Memory Comprehension Content recall “What were the names…”
Understanding Comprehension Concept grasp “What was the main conflict…”