What do crowd workers think about creative work?
What do crowd workers think about creative work?
Abstract
Crowdsourcing platforms are a powerful and convenient means for recruiting participants in online studies and collecting data from the crowd. As information work is being more and more automated by Machine Learning algorithms, creativity – that is, a human’s ability for divergent and convergent thinking – will play an increasingly important role on online crowdsourcing platforms. However, we lack insights into what crowd workers think about creative work. In studies in Human-Computer Interaction (HCI), the ability and willingness of the crowd to participate in creative work seems to be largely unquestioned. Insights into the workers' perspective are rare, but important, as they may inform the design of studies with higher validity. Given that creativity will play an increasingly important role in crowdsourcing, it is imperative to develop an understanding of how workers perceive creative work. In this paper, we summarize our recent worker-centered study of creative work on two general-purpose crowdsourcing platforms (Amazon Mechanical Turk and Prolific). Our study illuminates what creative work is like for crowd workers on these two crowdsourcing platforms. The work identifies several archetypal types of workers with different attitudes towards creative work, and discusses common pitfalls with creative work on crowdsourcing platforms.
Background
Workers on crowdsourcing platforms have become one of the most thoroughly studied sets of human subjects, and scientists have at their disposal a strong understanding of the demographics of workers on the crowdsourcing platforms. The academic literature primarily investigates crowdsourcing from the perspective of the requester of work, with a focus on optimizing costs, response times, and work flows. However, in studies in HCI, the ability and willingness of the crowd to contribute their creativity and participate in creative studies seems to be largely unquestioned. Our understanding of how workers perceive creative work is limited. Insights into the workers’ perspective are rare, but important, as they may inform the design of studies with higher validity. Given that creativity will play an increasingly important role in crowdsourcing due to human's ability to excel in areas that machines fall short, such as divergent thinking, recombination, and analogical transfer, it is imperative to develop an understanding of how crowd workers perceive creative work.
Approach
We launched a survey study on two crowdsourcing platforms: Amazon Mechanical Turk and Prolific. The two platforms are different in regard to worker demographics and what type of work is requested. We selected the two platforms to provide us a complementary insight into creative work on two of the most commonly used crowdsourcing platforms. The questionnaire included quantitative Likert-scale items as well as open-ended items asking the workers to elaborate on their answers. The questionnaire was completed by 323 workers of which 215 (102 Amazon Mechanical Turk workers and 113 Prolific workers) met our qualification criteria (approval rate of greater than 98%, 1000 completed tasks on Amazon Mechanical Turk, and prior encounters with creative work on the respective platform). Workers were paid USD 1 on Amazon Mechanical Turk and GBP 1 on Prolific. Later, we noticed our estimated task completion times being too low and compensated workers with bonuses to pay at least minimum wage (USD 7.50) to all workers. Three researchers open-coded all open-ended responses, allowing us to form a view of creative work on the platform and a sense for different archetypal worker profiles.
Results
Worker Archetypes
Professional workers largely prefer to avoid creative work, as – in their view – this type of work is precarious and connected with considerable uncertainty in the amount of time required to complete the task and the reward may not be worth the effort in the view of this worker. Workers reported of unfair rejections for subjective tasks. Collaborative work flows, as in many experiments focusing on group creativity, may cause delays in completing the work. The professional worker will try to avoid such tasks, as they cause idle time and forfeiting potential earnings from other tasks. Prior work found that professional workers complete the majority share of work on crowdsourcing platforms. Professional workers may thus have been exposed to a variety of different creative tasks on the crowdsourcing platforms, including standard psychometric tests, such as the Alternate Uses test. Foreknowledge and pre-exposure to such tasks may affect the validity of experiments and studies, as shown in the context of behavioral research.
Pragmatic workers may not care much about what type of tasks are given to them. In the view of this worker, requesters are going to request what they want, and some workers will be found who agree to work on the tasks. For this worker, it is important to have enough information about the task so that the worker can make an informed decision whether to accept the task or not.
Novelty seekers are inspired and attracted by novel tasks. The worker enjoys the unexpected and thus prefers working on creative tasks that the worker has never seen before. This may lead to a “tragedy of the commons” effect, where the worker quickly gets to know the different tasks and then becomes bored with the platform. Self-Developers are less motivated by monetary rewards, but by opportunities for learning something new or about themselves. Creative tasks, in this regard, are attractive to this worker, as they allow the worker to learn something about the worker’s own personality. For this type of worker, it is important to get post-task feedback in form of a debriefing with task results.
Casual workers are perhaps most open to creative tasks, as there is a chance that this type of worker has never completed such tasks before. On the other hand, as casual workers work only few hours per week (or even per month) on the platform, the workers may never come in contact with creative work and may think that there is no creative work on the crowdsourcing platform at all. This type of worker is also open to collaborative online experiments and workflows, as potentially precarious monetary rewards and task completion times do not play a strong role in the workers decision to start a task.
Workers' preference for working alone
Supporting creative work has been regarded as one of HCI’s grand challenges. One trend in current research of the field is supporting the creativity of groups. In our sample, we found clear evidence of workers shunning on collaborative work. Almost 90% of the participants preferred solo work and only 23 participants (10.7%) preferred collaborative work. We isolated several reasons for this preference. Enforced cooperation may cause issues both in time and rewards. Cooperation issues were often brought up, with productivity and efficiency being the second most popular reason for workers to prefer solo work. Yet, there is a small, but clear, group of workers who enjoy collaborative creative work. For instance, one Prolific worker noted: “The tasks that include others tend to be more exciting due to the anticipation of seeing how they will respond to each task.” Other workers informally noted that they simply enjoy working together with other people – a trait that, according to our study’s findings, is shared by only a minority of workers on the two crowdsourcing platforms
Pre-Exposure to Creative Work
Our study found evidence of worker nonnaïveté in regard to common creativity tests, as a subset of creative work requested on crowdsourcing platforms. The past encounters raise a point to consider: Is the data collected from these participants on creativity tests valid? Creativity tests measure a subject’s creativity, but may also be used to measure other constructs. Creativity tests – and especially those relying on an “a-ha” experience, such as Practical Insight Problems – may be affected by prior exposure. If a relatively large portion of a study’s sample has prior exposure to the very same creativity test, this should be considered in the study design and participant screening.
Learning from Creative Work
A subset of the workers mentioned learning as a motivator for completing creative work. The proportion of participants reporting on some aspect of learning taking place during creative work on the crowdsourcing platform was high. Of the 143 workers who elabo- rated on this question, only six (4.2%) reported not learning anything at all. About 30% of the participants (N = 43) mentioned that participation in creativity studies has led to discovering insights about their own personality. One fifth of the workers reported either developing their creative skills or awakening to the fact that they already are creative. Others mentioned having learned subject-specific skills from completing creative tasks. Increasing one’s productivity and concentrating for long periods of time were mentioned. Working on the platforms seems to encourage workers to engage and test their creativity in other areas, as exemplified by a worker who learned how to write scripts to support his work on the crowdsourcing platform.
Bibtex
@inproceedings{CHI2020,
title = {What do crowd workers think about creative work?},
author = {Jonas Oppenlaender and Aku Visuri and Kristy Milland and Panos Ipeirotis and Simo Hosio},
location = {Honolulu, HI, USA},
year = {2020},
numpages = {4 pages},
url = {http://jultika.oulu.fi/files/nbnfi-fe2020052538841.pdf},
booktitle = {Workshop on Worker-Centered Design: Expanding HCI Methods for Supporting Labor},
}