The impact of GenAI on critical thinking: how self-confidence shapes cognitive engagement

Challenge 

How does using AI tools affect the way people think critically at work? A team from Carnegie Mellon University and Microsoft Research wanted to understand when and how knowledge workers engage their critical thinking abilities while using AI. The research needed:

• A diverse group of professionals who regularly use AI tools at work
• Real examples showing exactly how people use AI in their daily tasks
• Clear data about when people stop to think critically about AI outputs
• Detailed insights into how confidence levels affect AI interactions
• Rich qualitative descriptions of thought processes and decision-making

Solution 

The research team used Prolific to recruit 319 knowledge workers who regularly use AI in their jobs. 

Quality participant matching 

Prolific connected the researchers with professionals across many industries who could share firsthand experiences with AI tools. Pre-screening capabilities meant that only participants with regular experience using AI at work were selected.

Reliable data collection 

The study gathered 936 detailed examples of real AI use cases. With only 14 responses needing exclusion for quality issues, Prolific's participants provided consistently thoughtful and thorough responses.

Flexible research setup 

Prolific integrated with the study's survey tools, letting researchers collect both structured ratings and open-ended explanations of how people think through their AI interactions.

Execution 

Participants each shared three different examples of using AI at work. For each, they described exactly what they were trying to achieve and explained how they used the AI tool. They rated their confidence in both themselves and the AI, detailed when they stopped to think critically, and shared what made them trust or question AI outputs. Then, they explained how their thinking process changed with AI.

Results 

The study revealed fascinating patterns in how people engage with AI:

• People who were more confident in AI tended to think less critically about it
• Those more confident in their own abilities did more critical thinking, even though it took more effort
• Critical thinking shifted from doing tasks directly to overseeing AI quality
• Workers found it harder to evaluate AI in areas where they lacked expertise

The findings also showed how critical thinking changes with AI:

• Instead of gathering information, people focus on verifying AI outputs
• Rather than solving problems directly, they guide and refine AI responses
• The emphasis moves from doing work to ensuring AI work meets standards

The research helps show what organizations need to watch for as AI becomes more common in workplaces. It highlights the importance of building both AI skills and critical thinking abilities.

Conclusion

For researchers studying how humans interact with AI, Prolific provided an ideal way to gather authentic examples and insights from real workplace settings. Researchers could explore complex cognitive behaviors while maintaining high data quality standards.

Citation: Lee, H., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. CHI '25, April 26-May 01, 2025, Yokohama, Japan.

Research institutions: Carnegie Mellon University and Microsoft Research.