The goal of this framework is to envision afuture of crowd work that cansupport more complex, creative, and highly valued work. At the highest level, a platformis needed for managing pools of tasksandworkers. Complex tasks must be decomposed into smaller subtasks, each designed with particular needs and characteristics which must be assignedto appropriate groups of workerswho themselves must be properly motivated, selected(e.g., through reputation), and organized (e.g., through hierarchy). Tasks may be structured through multi-stage workflowsin which workers may collaborateeither synchronously or asynchronously. As part of this, AImay guide (and be guided by) crowd workers. Finally, quality assuranceis needed to ensure each worker’s output is of high quality and fits together.

n human computation, people act as computational components and perform the work that AI systemslack the skillstocomplete

A promising approach that addresses some worker output issues examines the way that workers do their work rather than the output itself, using machine learning and/or visualization to predict the quality of a worker’s output from their behavior [119,120]

Of the research foci, quality control has arguably received the most attention so far. Approaches for quality control largely fall into two camps: up-front task design and post-hoc result analysis. Task design aimsto design tasks that are resistant to low-quality work.

Many tasks worth completing require cooperation –yet crowdsourcing has largely focused on independent work. Distributed teams have always facedchallenges in cultural differences and coordination[60], but crowd collaboration now must createrapport over much shorter timescales(e.g., one hour) and possibly wider culturalor socioeconomic gaps

Finally, it will be amajor research undertaking to invent and describe the tasks and techniques that succeed with synchronous collaboration

The two core challenges for realtime crowdsourcing will be 1) scaling upto increased demand for realtime workers, and 2) making workers efficient enough to collectively generateresultsahead of time deadlines.

Volunteer crowdsourcing platforms have evolved their own hierarchies and decision-making processes [104,156], appropriating techniques from other online communities where appropriate [101]. Most paid approaches have workers make hierarchical decisions collectively: for example, task decomposition and integration[75,80],quality oversight of each others’ contributions[78,100], and leader elections to represent collective opinions[83].

Complex tasks have dependencies, changing requirements, and require multiple types of expertise.

n the sections below, we survey and analyze the 12research focithat comprise our model. First, we consider the future of the work processesand how the workis organized and accomplished. Second, we consider the integration of crowd work and computation, including the symbiosis between human cognition, artificial intelligence(AI), and computationally-enabled crowd platforms. Finally, we consider crowd workers and how we can develop jobs, reputation systems, motivations, and incentives that will benefit them.

Unlike traditional organizations in which workers possess job security and managers can closely supervise and appropriately reward or sanction workers, or distributed computing systems in which processors are usually highly reliable, crowd work poses uniquechallenges for both workers and requestersranging fromjob satisfactiontodirection-setting, coordination, and quality control.

These same requirements exist in distributed computing, in which tasks need to be scheduled so that they can be completed in the correct sequence and in a timely manner, with data being transferred between computing elements appropriately.

However, crowd work can bea double-edged sword,equally capable of enhancing or diminishing the quality of workers’ lives.We maysee echoes of past labor abuses in globally distributed crowd work: extremely low pay for labor,with marketplaces such as Amazon’s Mechanical

We focus this paper on paid,onlinecrowd work, which we define here as the performance of tasks online by distributed crowd workers who are financially compensated by requesters(individuals, groups, or organizations). In this sense, crowd work is a socio-technical work systemconstituted through a set of relationships that connect organizations, individuals, technologies and work activities