Exploring the goals and challenges of crowdworkers: An interview with Dr. Tahir Abbas

Goal setting is a proven technique to improve motivation, performance, and learning in many areas of life. Recently, researchers have started exploring how goal setting might benefit crowdworkers who contribute to research on various online platforms. However, little is known about the specific goals that crowdworkers set for themselves and the challenges they face in achieving them.

I recently had the opportunity to interview Dr. Tahir Abbas, the author of a new study that provides fascinating insights into crowdworkers’ goal-setting practices by surveying over 200 workers on Amazon Mechanical Turk and Prolific about the types of goals they set, the strategies they use, the obstacles they encounter, and the tools they utilize.

You can access the full paper here.

Can you tell me a little bit about yourself and your area of research?

I’m currently working as a postdoc researcher at TU Delft University, engaged in different types of user research including goal-setting behaviors. I’m interested in how goal setting can help to improve people’s achievements and I have specifically looked at this through the lens of crowdsourcing.

I completed my PhD in Human Computer Interaction from Eindhoven University of Technology, where I was also heavily involved in using crowdsourcing for research. One of the areas that I worked on was building crowd-powered conversational applications - leveraging human intelligence to help make AI systems even more intelligent, resilient, and robust, in situations where it typically fails.

We’re here today to talk about your study that looked into the goal-setting behaviors of crowd-sourced participants. Can you define what goal-setting actually refers to?

Goal-setting for me is essentially like a list of tasks or a mission that you want to achieve in the future, whether that’s a week, a month, a year, or longer. It's a kind of obligation that you put on yourself to make yourself accountable. Some people write tasks on their to-do list and try to stick with them, whereas some people will use smart technologies like mobile devices to write their goals. So for me it’s a kind of obligation or mission that people create for themselves to make the task more measurable and specific.

Why do you think it is important to understand these behaviors?

It is a very under-researched area currently. Most of the research that has been done in this space that uses goal-setting theory has been conducted with the aim of improving the performance of a task by crowdsourced participants. But one thing that is missing was a comprehensive understanding of what types of goals these workers have, whether there are any obstacles or barriers that they experience when them, and if they use existing technologies or tools.

All of these questions were unanswered and that's why we decided to delve into this. Additionally, companies are now able to create more useful, engaging tools for crowdsourced workers, to improve their performance and also maybe to improve their well-being. So this area of research could also improve existing crowdsourcing platforms, their workflows, and the types of strategies they use to manage their workers.

Can you tell me a little bit about how you designed the study?

Initially, we chose to use three different crowd sourcing platforms, Prolific, MTurk, and Toloka. However, it was too challenging to set up the study on Toloka so we decided to go with just two. Based on our literature review I designed four over-arching research questions:

(1) What type of goals do workers set for themselves, and why?

(2) How often and for how long do workers set goals?

(3) What do crowd workers do to achieve their goals and what potential barriers prevent this?

(4) How can we use technology to assist workers in achieving their goals?

I then designed a survey in Qualtrics to elicit participants’ thoughts on these areas in a fully qualitative design. In total, we recruited 205 participants, 100 from Prolific and 105 from MTurk. The analysis of the qualitative data was performed using the Dedoose software and I coded all of the open-ended questions to identify major themes and concepts that emerged.

How did you choose the platforms? Why MTurk and Prolific?

MTurk was used because of its popularity and because it's very old. Most researchers who use crowdsourcing approaches are familiar with MTurk so it was important to study the goal-setting behavior of its workers.

We choose Prolific because it's gaining a lot of popularity in the academic setting due to its transparency and high-quality data. This is also my personal experience that in the past studies that I have done on Prolific, I found that the quality was very high. The workers were very trustworthy and I got better qualitative data compared to other crowdsourcing platforms.

There are a lot of results in your paper, but could you give me a brief recap of the main findings?

Regarding the types of goals workers set for themselves, we found that income-related goals were very common and most participants appeared to be extrinsically motivated. But interestingly we also found that some participants are intrinsically motivated and are motivated by a desire to help researchers to conduct their research or to give quality feedback. There were some also workers who wanted to use these platforms just to meet their expenses.

As for goal attainment strategies, we found that workers use different types of media or tools, such as to-do lists, journals, diaries, or whatever they can as a reminder to track their progress, recall their goals, and hold themselves accountable for their success.

In terms of the potential barriers, we found that one of the biggest barriers was a busy lifestyle. Workers simply didn’t have the time to really invest in crowdsourced work to the degree that they wanted. Another big barrier was distraction, procrastination, or feeling overwhelmed.

Interestingly, we also received quite a high number of responses that flagged either depression or ongoing medical issues as being a barrier to achieving their crowdsourcing goals. Finally, workers also cited goal-setting software flaws as being problematic, with most being clumsy and difficult to use, prone to error, or having a lack of personalization features.

In relation to how workers use technology to assist with goal-setting, many mentioned intelligent reminder applications that can proactively motivate them and keep them encouraged throughout their day. We also observed that many participants stated that goal-setting technologies that had game-like features (like rewards or points) and were customizable we much preferred to other, more simplistic tools.

Given that you sampled from both Prolific and MTurk, did you notice any clear difference between the two samples in terms of their behaviors?

Yes, there were a number of differences that we observed. First, Prolific workers created more goals than those on MTurk, while MTurk users typically set long-term goals than those on Prolific. This might be partly due to the fact that MTurk has been around for a longer time, which means that its workers are used to longer-term goal-setting.

Another clear difference was that Prolific workers were more likely to set giving quality output as a goal, and also used it as a secondary source of income rather than the primary source of income. MTurk workers mentioned making money as a primary goal considerably more often than those on Prolific.

What are the key takeaways here for both platform developers and researchers who are running their studies online?

I think it is equally important for both platform developers and researchers that can use this information to investigate how to translate these results into actionable strategies to support various goal types and motivation of crowdsourced workers.

One big takeaway from this work is that we found a certain number of workers who have some sort of disability, such as a medical or mental health issue. I think this suggests that platforms should begin to think about how to adapt their approach to provide more flexible working conditions for these individuals, an area that is currently lacking. So often we have certain rules and regulations when it comes to the physical workplace, for example providing extra time to students who have a disability, and I think that similar mechanisms could be deployed on crowdsourcing platforms to make them more inclusive.

From a researcher perspective, we often think that workers are only strongly motivated by monetary gains, but these data showed that was not the case. We found that workers use these platforms for various reasons unrelated to financial gain, and researchers should be aware of that when choosing where and how to run their studies.

What do you think are some future avenues of research in this area?

If given the chance to extend this study, I would focus on the idea of inclusivity. This area has been under-researched and has not received much attention. For instance, if we focus on the mental health of crowd workers, there are many opportunities for researchers to explore such as designing interventions to help these workers reduce stress and make their tasks more enjoyable.

One of my colleagues is also working on the idea of a sustainable crowdsourcing environment, exploring whether interventions or strategies can be created to help workers maintain physical health. For instance, when working on a task that lasts an hour or longer, it is common to remain in the same position without moving, which is not healthy for the individual. Instead, he's actually proposing a gesture-based approach to improve health. For example, what if a simple task can be completed using gestures rather than key presses? The fast advancement of AI models means that we can start to think about these kinds of options in more depth.

You can keep up to date with Dr. Abbas’ work via his LinkedIn.

This research was carried out using the Prolific platform and our reliable, highly engaged participants. Sign up today and conduct your best research with our powerful and flexible tools.