Using Prolific to Estimate the Accuracy of Self-Reported data

MeasuringU is a mixed-methods research and software firm based in Denver, CO. They offer both quantitative and qualitative support for all stages of customer and user research.

The task 

How much did you spend last week? 

If you had to provide receipts of proof of purchases, how accurate do you think your estimate would be? 

It can be quite difficult (and invasive) to track actual behaviors and activities. Consequently, many studies rely on self-reporting for data such as purchase history and health information. But how accurate is self-reported data?

In 2019, MeasuringU gathered and assessed self-reports of recent data to understand the following:

1. Are self reported purchar rates accurate?
2. Are self-reported purchase amounts accurate?
3. Do better experiences precede more transactions?

The data were collected in two-parts longitudinal study taken on prolific in which participants (1) reported their purchase behaviour and (2) subsequently provided supporting receipts or records. 

The challenge

In the first survey, a list of well-known mass merchant websites (e.g Walmart, Best Buy, Target) was presented to participants. Each participant selected all the websites from which they had made a purchase in the last 90 days. One of the selected websites was randomly chosen and, for that website, the participant was asked how many purchases were made and how much was spent over the 90-day period. 

For the second survey, MeasuringU followed up with participants who were willing to provide receipts or records verifying the transaction data. 

As such, there were several challenges with the recruit for this study due to: 

1. Needing trustworthy respondents who could recall purchase dates and amounts for a variety of products from the past 90 days 
2. Needing participants willing to provide personal receipts or records verifying the transaction data
3. Needing the ability to invite eligible respondents for a follow-up survey

The solution

Prolific was the perfect platform to handle the recruit for this self-reported longitudinal study given the platform's capabilities as well as their history of high data quality and low drop-off rates. 

MeasuringU received self-reported purchase data from 1,160 participants. A bit less than half (515 participants, about 44%) indicated that they would be willing to participate in the follow-up survey. 

They then invited roughly half (237 of the 515) of the respondents who agreed to be contacted to continue participating. In the second survey participants were to upload receipts and records that would provide evidence of the purchases. 

Of the 237 invited, 118 (50%) completed the survey with at least one receipt uploaded. Along with receipts, several left illuminating comments which shed some light on the differences between self-report and records:

The results

MeasuringU analysis of self-reported purchase data compared to verified receipt data found: 

Purchase rates were accurate. 
Almost all participants (98%) who completed the study were able to provide proof of purchase for the website they reported purchasing from. This suggests that self-reported purchase rates are highly likely to be accurate. That is, if a respondent said they purchased from eBay or Walmart, they most likely did. 

Amount spent was less accurate. 
Only 29% of the receipts that matched the store had amount totals that were within 10% of the self-reported amount and 44% of receipts were within 20% of the amount. Only 22% of receipts that matched the store purchase and the 90-day timeframe were within 10% of the self-reported amount.  

This low accuracy is likely a consequence of at least three things: inaccurate recall from smaller non-reoccurring purchases, telescoping the purchases into a more recent time period (leading to overestimating amounts), and the consequence of respondents estimating amounts without being asked to check or keep records of the transactions.  

Using the verified purchase amounts did show a similar although much more attenuated, pattern of higher purchase amounts for participants with the best experience. This suggests an “unaudited” estimate of small amounts spent on a website may be at best a crude indicator of the actual amount spent. It also suggests that a 20% threshold may be more realistic for self-reported purchase accuracy.

Better experiences portend more transactions.
Using the total number of receipts participants uploaded as a proxy for transactions and comparing that with concurrently collected SUPR-Q® scores (ratings of the quality of website experiences - usability, credibility, apperance, and loyalty) suggests that better experiences lead to more purchases.