- Aug 2018
The goal of this framework is to envision afuture of crowd work that cansupport more complex, creative, and highly valued work. At the highest level, a platformis needed for managing pools of tasksandworkers. Complex tasks must be decomposed into smaller subtasks, each designed with particular needs and characteristics which must be assignedto appropriate groups of workerswho themselves must be properly motivated, selected(e.g., through reputation), and organized (e.g., through hierarchy). Tasks may be structured through multi-stage workflowsin which workers may collaborateeither synchronously or asynchronously. As part of this, AImay guide (and be guided by) crowd workers. Finally, quality assuranceis needed to ensure each worker’s output is of high quality and fits together.
Proposed framework to address crowdwork management challenges: shared resources, relationships, and crowd labor.
n human computation, people act as computational components and perform the work that AI systemslack the skillstocomplete
Human computation definition.
A promising approach that addresses some worker output issues examines the way that workers do their work rather than the output itself, using machine learning and/or visualization to predict the quality of a worker’s output from their behavior [119,120]
This process improvement idea has some interesting design implications for improving temporal qualities of SBTF data: • How is the volunteer thinking about time? • Where does temporality enter into the data collection workflow? • What metadata do they rely on? • What is their temporal sensemaking approach?
Of the research foci, quality control has arguably received the most attention so far. Approaches for quality control largely fall into two camps: up-front task design and post-hoc result analysis. Task design aimsto design tasks that are resistant to low-quality work.
Quality control processes is definitely a tension for SBTF.
A better integrated task design and verification process at the end of activations could be more effectively address information quality concerns.
Many tasks worth completing require cooperation –yet crowdsourcing has largely focused on independent work. Distributed teams have always facedchallenges in cultural differences and coordination, but crowd collaboration now must createrapport over much shorter timescales(e.g., one hour) and possibly wider culturalor socioeconomic gaps
In Kittur's example, synchronous collaboration describes a temporal aspect (timescale and tempo of the work) related to how the collaboration is structured or not.
"Short periods of intense crowd collaboration call for fast teambuilding and may require the automatic assignment of group members to maximize collective intelligence."
Finally, it will be amajor research undertaking to invent and describe the tasks and techniques that succeed with synchronous collaboration
Could this be a theme of the SBTF time study?
The two core challenges for realtime crowdsourcing will be 1) scaling upto increased demand for realtime workers, and 2) making workers efficient enough to collectively generateresultsahead of time deadlines.
One aspect of temporality in Kittur's study is related to "realtime" which they describe as the time need to scale up workers and efficiency speed of workers.
The other temporal aspect is synchronicity of workers.
Volunteer crowdsourcing platforms have evolved their own hierarchies and decision-making processes [104,156], appropriating techniques from other online communities where appropriate . Most paid approaches have workers make hierarchical decisions collectively: for example, task decomposition and integration[75,80],quality oversight of each others’ contributions[78,100], and leader elections to represent collective opinions.
Examples of hierarchical decision-making by both volunteer and paid crowd workers.
Complex tasks have dependencies, changing requirements, and require multiple types of expertise.
Characteristics of complex crowd work.
Later, Kittur refers to complex crowd work as those involving "creativity, brainstorming, essay writing, music composition or civic planning."
Temporality is definitely a work flow issue for SBTF.
However, "realtime" is the only temporal attribute noted in this study but it seems to relate only to completion speed and present/immediacy of tasks.
n the sections below, we survey and analyze the 12research focithat comprise our model. First, we consider the future of the work processesand how the workis organized and accomplished. Second, we consider the integration of crowd work and computation, including the symbiosis between human cognition, artificial intelligence(AI), and computationally-enabled crowd platforms. Finally, we consider crowd workers and how we can develop jobs, reputation systems, motivations, and incentives that will benefit them.
Crowd work processes: Workflow, task assignment, hierarchy, realtime crowd work, synchronous collaboration. quality control
Crowd computation: Crowds guiding AI, AIs guiding crowds, crowdsourcing platforms
Crowd worker future: Job design, reputation/credentials, motivation/rewards
Unlike traditional organizations in which workers possess job security and managers can closely supervise and appropriately reward or sanction workers, or distributed computing systems in which processors are usually highly reliable, crowd work poses uniquechallenges for both workers and requestersranging fromjob satisfactiontodirection-setting, coordination, and quality control.
In the literature, quality tends to be used as an attribute of the output (content, HIT, etc.) but could/should it also refer to the crowd worker experience, as Kittur notes: "job satisfaction, direction-setting, coordination, and quality control"?
How are these factors incorporated into the process and incentive system?
These same requirements exist in distributed computing, in which tasks need to be scheduled so that they can be completed in the correct sequence and in a timely manner, with data being transferred between computing elements appropriately.
time factors in crowd work include speed, scheduling, and sequencing
However, crowd work can bea double-edged sword,equally capable of enhancing or diminishing the quality of workers’ lives.We maysee echoes of past labor abuses in globally distributed crowd work: extremely low pay for labor,with marketplaces such as Amazon’s Mechanical
Crowd work offers flexibility to both workers and requesters to overcome labor shortages, need for expertise, and geographic boundaries.
However, they are very real concerns about exploitation, piecemeal wages, unethical/dubious work, emphasis on speed over quality, and dehumanizing work conditions.
We focus this paper on paid,onlinecrowd work, which we define here as the performance of tasks online by distributed crowd workers who are financially compensated by requesters(individuals, groups, or organizations). In this sense, crowd work is a socio-technical work systemconstituted through a set of relationships that connect organizations, individuals, technologies and work activities
Kittur's definition of crowd work:
"...performance of online tasks by crowd workers who are financially compensated by requesters."
- synchronous collaboration
- social coordination
- temporal design
- clock time
- design implications
- design implication
- crowd work
- human computation