Super-intelligence or superstition? How Prolific empowered cutting-edge AI research

Challenge 

When an AI predicts your behavior, what makes you trust it? Researchers from MIT Media Lab asked the same question and sought to find out the psychological drivers behind trust in AI predictions by comparing them with beliefs in astrology and personality tests. The study's complexity, however, posed several challenges:

• Recruiting a diverse, reliable participant pool to make sure the data is high quality and representative.
• Designing a framework that combined questionnaires, behavioral tasks, and simulated decision-making.
• Ensuring data quality for high-stakes analysis, where unreliable responses could jeopardise the findings.

Solution 

The researchers turned to Prolific to recruit 238 participants, benefiting from the platform’s unique advantages:

Diverse and targeted recruitment

Prolific’s marketplace allowed the researchers to source participants across a wide range of demographics, including age, gender, and education levels for a representative dataset. This was necessary for exploring how variables like attitudes toward AI and beliefs in the paranormal influenced trust.

High-quality data

Only 62 out of 300 initial responses were discarded, a testament to Prolific’s ability to deliver reliable participants. Compared to industry benchmarks, the low exclusion rate underscores Prolific’s ongoing investment in data quality.

Ease of integration

Prolific's compatibility with external tools like Qualtrics enabled seamless execution of the study, from belief assessments to behavioral tasks. Such flexibility allowed the researchers to focus on their experimental design without technical barriers.

Execution 

Participants completed detailed questionnaires exploring their beliefs and personality traits. They then engaged in a simulated investment game, where they provided an interactive basis for personalised predictions. 

These deliberately fictitious predictions were presented as either positive (highlighting rational decision-making) or negative (suggesting impulsivity). 

Results 

The study revealed fascinating insights into the psychology of trust in predictions:

• Participants who believed in astrology or personality-based predictions were more likely to trust AI predictions.
• Positive attitudes toward AI and belief in paranormal phenomena increased perceptions of AI predictions’ validity, reliability, and usefulness.
• Analytical thinking skills had little to no impact on trust in predictions.
• Conscientious individuals were more sceptical of predictions from all sources.
• Greater interest in the prediction topic led to higher trust, regardless of whether the source was AI, astrology, or personality analysis.

Conclusion 

The study, conducted by MIT Media Lab, Microsoft Research, and the Massachusetts Institute of Technology, sheds light on how psychological traits influence trust in AI predictions. Prolific played an essential role in overcoming recruitment and data quality challenges, demonstrating its value for high-stakes, cutting-edge research.

For researchers tackling complex studies, Prolific offers proven solutions with a diverse participant pool, seamless integration with research tools, and industry-leading data quality.

Citation Lee, E., Pataranutaporn, P., Amores, J., & Maes, P. (2024). Super-intelligence or superstition? Exploring psychological factors underlying unwarranted belief in AI predictions. arXiv:2408.06602

Research institutions: ​​MIT Media Lab, Microsoft Research, and Massachusetts Institute of Technology