Data Model MEGAnno+ extends MEGAnno's data model where data Record, Label, Annotation, Metadata (e.g., text embedding or confidence score) are persisted in the service database along with the task Schema.

To implement our system as an extension to MEGAnno (Zhang et al., 2022), an in-notebook exploratory annotation tool.

MEGAnno+ is designed to provide a convenient and robust workflow for users to utilize LLMs in text annotation. To use our tool, users operate within their Jupyter notebook (Kluyver et al., 2016) with the MEGAnno+ client installed.

LLM annotators and human annotators should not be treated the same, and annotation tools should carefully design their data models and workflows to accommodate both types of annotators.

we go beyond using LLMs to assist annotation for human annotators or to replace human annotators. Rather, MEGAnno+ advocates for a collaboration between humans and LLMs with our dedicated system design and annotation-verification workflows.

Despite these advancements, it is essential to acknowledge that LLMs have limitations, necessitating human intervention in the data annotation process. One challenge is that the performance of LLMs varies extensively across different tasks, datasets, and labels. LLMs often struggle to comprehend subtle nuances or contexts in natural language, making involvement of humans with social and cultural understanding or domain expertise crucial.

Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels.

Valarie A Zeithaml and William L Fuerst. 1983. Age differences in response to grocery store price information. Journal of consumer affairs 17, 2 (1983), 402–420.

Mary E Sesto, Curtis B Irwin, Karen B Chen, Amrish O Chourasia, and Douglas A Wiegmann. 2012. Effect of touch screen button size and spacing on touch characteristics of users with and without disabilities. Human Factors: The Journal of the Human Factors and Ergonomics Society 54, 3 (2012), 425–436.

Zhao Xia Jin, Tom Plocher, and Liana Kiff. 2007. Touch screen user interfaces for older adults: button size and spacing. In Universal acess in human computer interaction. coping with diversity. Springer, 933–941.

Robin Brewer, Raymundo Cornejo Garcia, Tedmond Schwaba, Darren Gergle, and Anne Marie Piper. 2016. Exploring Traditional Phones as an E-Mail Interface for Older Adults. ACM Transactions on Accessible Computing (TACCESS) 8, 2 (2016), 6.

Janan Al-Awar Smither and Curt C Braun. 1994. Technology and older adults: Factors affecting the adoption of automatic teller machines. The Journal of General Psychology 121, 4 (1994), 381–389.

Wiktoria Wilkowska and Martina Ziefle. 2009. Which factors form older adults' acceptance of mobile information and communication technologies? Springer.

Kerryellen G Vroman, Sajay Arthanat, and Catherine Lysack. 2015. "Who over 65 is online?" Older adults' dispositions toward information communication technology. Computers in Human Behavior 43 (2015), 156–166.

Phil Turner, Susan Turner, and Guy Van de Walle. 2007. How older people account for their experiences with interactive technology. Behaviour & Information Technology 26, 4 (2007), 287–296.

Hironobu Takagi, Akihiro Kosugi, Tatsuya Ishihara, and Kentarou Fukuda. 2014. Remote IT education for senior citizens. In Proceedings of the 11th Web for All Conference. ACM, 41.

Karen Renaud and Judy Van Biljon. 2008. Predicting technology acceptance and adoption by the elderly: a qualitative study. In Proceedings of the 2008 annual research conference of the South African Institute of Computer Scientists and Information Technologists on IT research in developing countries: riding the wave of technology. ACM, 210–219.

Chee Wei Phang, Juliana Sutanto, Atreyi Kankanhalli, Yan Li, Bernard CY Tan, and Hock-Hai Teo. 2006. Senior citizens' acceptance of information systems: A study in the context of e-government services. Engineering Management, IEEE Transactions on 53, 4 (2006), 555–569.

Bjorn Niehaves and Ralf Plattfaut. 2014. Internet adoption by the elderly: employing IS technology acceptance theories for understanding the age-related digital divide. European Journal of Information Systems 23, 6 (2014), 708–726.

HH Nap and HP de Greef. 2010. Self-efficacy & stress in senior computer interaction. In Proceedings of the 28th Annual European Conference on Cognitive Ergonomics. ACM, 227–230.

Michael G Morris and Viswanath Venkatesh. 2000. Age differences in technology adoption decisions: Implications for a changing work force. Personnel psychology 53, 2 (2000), 375–403.

Tracy L Mitzner, Wendy A Rogers, Arthur D Fisk, Walter R Boot, Neil Charness, Sara J Czaja, and Joseph Sharit. 2014. Predicting older adults' perceptions about a computer system designed for seniors. Universal Access in the Information Society (2014), 1–10.

Chaiwoo Lee and Joseph F Coughlin. 2014. PERSPECTIVE: Older Adults' Adoption of Technology: An Integrated Approach to Identifying Determinants and Barriers. Journal of Product Innovation Management (2014).

Sri Kurniawan. 2008. Older people and mobile phones: A multi-method investigation. International Journal of Human-Computer Studies 66, 12 (2008), 889–901.

Vicki L Hanson. 2011. Technology skill and age: what will be the same 20 years from now? Universal Access in the Information Society 10, 4 (2011), 443–452.

Mary C Gilly and Valarie A Zeithaml. 1985. The elderly consumer and adoption of technologies. Journal of consumer research (1985), 353–357.

Nancy M Gell, Dori E Rosenberg, George Demiris, Andrea Z LaCroix, and Kushang V Patel. 2013. Patterns of technology use among older adults with and without disabilities. The Gerontologist (2013), gnt166.

Helene Gelderblom, Tobie van Dyk, and Judy van Biljon. 2010. Mobile phone adoption: Do existing models adequately capture the actual usage of older adults?. In Proceedings of the 2010 annual research conference of the south african institute of computer scientists and information technologists. ACM, 67–74.

Arthur D Fisk, Wendy A Rogers, Neil Charness, Sara J Czaja, and Joseph Sharit. 2009. Designing for older adults: Principles and creative human factors approaches. CRC press.

Anna Dickinson, Alan F Newell, Michael J Smith, and Robin L Hill. 2005. Introducing the Internet to the over-60s: Developing an email system for older novice computer users. Interacting with Computers 17, 6 (2005), 621–642.

Mario Conci, Fabio Pianesi, and Massimo Zancanaro. 2009. Useful, social and enjoyable: Mobile phone adoption by older people. In Human-Computer Interaction–INTERACT 2009. Springer, 63–76.

Miha Cimperman, Maja Makovec Brenčič, Peter Trkman, and Mateja de Leonni Stanonik. 2013. Older adults' perceptions of home telehealth services. Telemedicine and e-Health 19, 10 (2013), 786–790.

Luca Buccoliero and Elena Bellio. 2014. The adoption of silver e-Health technologies: first hints on technology acceptance factors for elderly in Italy. In Proceedings of the 8th International Conference on Theory and Practice of Electronic Governance. ACM, 304–307.

Today's generations of older adults have not grown up with information and communications technologies that are widely available these days. Thus, there is "a natural confound of age and experience, since today's older adults are exposed to these technologies at a different point in their lives than today's young adults." [17]

Older people are less likely to have peers with sufficient technology experiences compared to their younger counterparts.

Incorporating these human factors and practical design suggestions for older adults, Fisk et al. proposed key recommendations for designing mobile devices for this age group [12].

Studies have shown that typical interaction components and techniques of a smartphone often prevent older adults from smooth and instant interactions with it. For example, the small size and the low contrast of buttons on a mobile display has a significant negative influence on interaction performance such as speed and accuracy [18], and decline in motor skills is correlated with time required to complete a task [30].

Lee and Coughlin reviewed studies of older adults' technology acceptance and identified ten factors that are critical facilitators or determinants of older adults' acceptance of technology: value, usability, affordability, accessibility, technical support, social support, emotion, independence, experience, and confidence [20].

most works point out that an individual's personal context [38] and the social context [36] in which the technology is introduced are the primary factors influencing the perception of, experience with, and evaluation of new technological developments among older adults [19].

One exception is the senior technology acceptance model (STAM) [28]. Using TAM, UTAUT, and several other works as theoretical underpinning, Renaud and Biljon proposed a model to explain older adults' mobile phone adoption.

Several studies have attempted to determine older adults' acceptance of technologies in general, and healthcare-related systems in particular, using the UTAUT framework. (e.g., email [14], a telehealth service [7]).

As a result, older adults and their adoption of new technologies have been a topic of active research since the advent of consumer technologies (e.g., automated teller machine [32], scanner-equipped grocery stores [41], electronic funds transfer [15]).

Seniors have historically been late adopters to the world of technology compared to their younger counterparts [24, 40].

Nowadays, older adults are increasingly adopting and adapting to information and communication technologies [5].

smartphone ownership among older adults has significantly risen in recent years [3]. However, its adoption levels among older adults in the US still sit at 27% as of 2015, whereas some 85% of Americans aged 18-29 are smartphone owners [31].

Lastly, while our findings are based on only 24 participants, the sample size is commensurate with the Ground Theory approach.

We analyzed the second-round interview data using inductive and deductive approaches informed by grounded theory and other qualitative analysis methods [33, 22].

Components in red boldface in Figure 3 provide a preview of the new elements we have identified and their relationship to the components proposed in earlier models.

Triangulating the empirical findings from our preliminary results with the existing theoretical models, we proposed an extension of the existing theoretical models that explains the technology acceptance behavior of our participants who were aged 60 or over. Our proposed model incorporates key elements of prior models and introduces novel components that significantly influence the participants' technology acceptance, namely one new phase, intention to learn, and three factors, self-efficacy, conversion readiness and peer support.

Consolidating our preliminary findings with the existing models, we propose an extended technology acceptance model for older adults illustrated in Figure 3. Extending to the predecessor theories, our tentative model introduces the perceived effort of learning a new technology as an obstacle for older adults' technology acceptance, which has not been reported in any studies of younger adults' technology acceptance.

In particular, we identified an additional phase that is prominent among the participants, intention to learn, but did not appear in prior models. Then, we identified three new factors that significantly influence their technology acceptance but which are, again, not represented in the existing models: self-efficacy, conversion readiness, and peer support.

The beauty of the GP-TSM technique lies in its simplicity: at its core, all GP-TSM does is change the visual saliency of words by adjusting their opacity. This preserves the integrity of the original text and minimizes "ergonomic obtrusiveness" [100] while providing readers with a form of "contextual cuing" to arm them with "incidental knowledge about global context", which they can harness to better assign visual attention and memory when reading [40].

Furthermore, according to Stevens's power law, people perceive changes in gray scale not linearly, but rather by a factor of approximately 0.5 [71]. For instance, a threefold increase in opacity might only be perceived as 1.5 times more significant, further complicating the differentiation of levels.

This sequence resonates with efficient content absorption strategies highlighted in speed reading literature, where readers first capture the gist and then delve deeper [1, 63]. The interface, therefore, may inadvertently facilitate this structured, layered reading approach, which might explain the improvement in reading efficiency and comprehension.

We adopt the term "saliency" based on its definition (a "bottom-up, stimulus-driven perceptual quality which makes some items stand out from their neighbors") [42], and its use in augmented reality [85, 88], computer vision [17, 55], and cognitive science [37, 56].

Modulating text saliency is a widely studied aspect of textual information representation. This technique modifies the visual attributes of text to promote words of interest and guide readers' attention, making pertinent information more perceptible and thereby enhancing comprehension and the user experience [12, 42].

compressive summarization aims to select the shortest subsequence of words within a sentence that yields an informative and grammatical sentence [64]. This framework allows for a more concise representation of the original content while retaining the essence of its meaning.

Given the cognitive effort reading requires, readers frequently resort to skimming, which is a rapid, selective, and non-linear form of reading [2]. Eye tracking studies [30, 74] validate that such behavior is extremely common. However, multiple studies have suggested a significant trade-off between reading speed and comprehension [65, 66, 76, 87].

Specifically, automated summarization methods can introduce multiple types of errors: "crimes" of omission, hallucination, and misrepresentation.

Automated text summarization techniques, including but not limited to crowd-powered systems [10], prompting large language models (LLMs) [105], and other AI technologies, can address a subset of these difficulties, i.e., the resulting text may be shorter, with simpler sentence structures and fewer unusual words [62]. However, unless there is information within the original document that is truly redundant, the result is a lossy representation of the original document, regardless of whether the process is abstractive or extractive.

Our goal is to modulate the saliency of words in the original text so that users can easily bypass certain words during skimming while maintaining an uninterrupted reading flow.

Be resilient to AI errors by enabling the reader to (a) notice, (b) have enough context to judge, and (c) easily recover from, automated decisions they disagree with.

Support skimming without interrupting flow. The system should improve skimming of text while minimizing the impact on the user's natural reading flow. In particular, as much as possible, it should avoid presenting users with salient text that is unparsable as a coherent thought, i.e., the system should present a complete sentence rather than a phrase or sentence fragment.

Support reading at multiple levels of detail. The system should help users navigate the full complexity of a text, shifting focus seamlessly between different levels of semantic coverage, or granularity, from the big picture to the fine details.

Integrate seamlessly into existing reading experiences. The system should complement and not interfere with the existing digital reading workflow that people are used to. It should provide all the functionalities in the same view, minimizing the overhead of mode and context switching.

Remain faithful to the original text. The system should not automatically reword or add new words or phrases to the original text. It should preserve the original text, while rendering it in a way that aids reading, skimming, or information retrieval.

We aspired to design a text rendering interface that alleviates some of the cognitive demands of reading, skimming, or performing information retrieval on natural language documents—particularly those with long, complicated sentences—without compromising the integrity of the original content.

One prior piece of HCI work, ParaLib [51], does explicitly exploit these theories for system feature design, but does this in the domain of code.

There are two hypothesized benefits of this view. One is based on an understanding of human perception: the grid layout should help users compare more LLM responses because the spatial arrangement assists their memory. The other benefit is based on Variation Theory, which posits that discerning the impact of a critical aspect, for example model temperature, is only possible when experiencing variation along that dimension, isolated from variation along other dimensions.

Given that the features implemented in this work are in line with design implications of Variation Theory and Analogical Learning Theory, the results suggest that there may be further utility of these theories for guiding the design of future systems that help users make sense of data and form mental models from examples.

Theories of human concept learning suggest that a key step in forming accurate, robust mental models of a phenomenon is to be able to discern the underlying dimensions of variation (Variation Theory) and any latent structures beneath superficial details (Analogical Learning Theory). By detecting and communicating which sentences are both structurally analogous (by virtue of their position within the response) and semantically related (by virtue of highly overlapping content), users should be able to more easily identify emergent structures, as well as compare and contrast particular compositions of structural elements across responses and syntactic elements that may vary in meaningful ways across analogous sentences within those responses. These theories assert that these subtasks are key ingredients in forming those robust accurate mental models, i.e., learning from the LLM responses in order to better perform their overarching task.

In this work, in line with Variation Theory, the existing and novel features instantiated and described in the next subsection collectively identify patterns of consistency, variation, or both; they are explicitly designed to make emergent dimensions of consistency and variation easier for the user to perceive.

Variation Theory describes how helping people perceive the different dimensions of consistency and variation across examples (here, LLM responses) of the object of learning helps them more quickly and robustly leap to more accurate mental models. Analogical Learning Theory describes how people can form mental models or schema from perceiving structural analogical relationships across superficially varying examples (again, here LLM responses).

Variation Theory [35] and Analogical Learning Theory [13, 14] each propose mechanisms for how people may conceive and update their mental models based on concrete examples, or use their mental model in new situations.

participants seemed to prefer engaging with the text directly without having to articulate a lens with which to look at the corpus, since their analysis goal may be initially under-defined.

we want to decorate text to show pre-computed relationships, such as string matches or analogous sentences, across responses. In this way, we help users shift cognitive bandwidth away from identifying overlapping or \

In our formative study, we found that automated analysis rarely captured what the participants were looking for when inspecting LLM responses.

We want to support a wide range of tasks that involve sensemaking. For example, we want to support the detection of similarities and differences between individual responses as well as groups of responses, and support the detection of

We aim to make 10s to 100s of LLM responses cognitively comfortable to peruse, as this was the scale we found to be most relavent in our formative study.

No one person on the Excel team is focused on the macro recorder.

The macro recorder doesn't work!

Four tips for using the macro recorder

Never use AutoSum or Quick Analysis while recording a macro.

Assigning a macro to a form control, text box, or shape

Creating a macro button on the Quick Access Toolbar

Creating a macro button on the ribbon

Running a macro

Filling out the Record Macro dialog box

Overview of recording, storing, and running a macro

As corporate IT departments have found themselves with long backlogs of requests, Excel users have discovered that they can produce the reports needed to run their businesses themselves using the macro language Visual Basic for Applications (VBA).

VBA enables you to achieve tremendous efficiencies in your day-to-day use of Excel. VBA helps you figure out how to import data and produce reports in Excel so that you don't have to wait for the IT department to help you.

Eye-typing forces users to think in terms of individual letters. This has a cognitive cost and is not a fluid means of communication.

To use such a switch for typing, the SGD interface must be designed with this in mind from the beginning. The most common solution is a scanning keyboard.

TTF refers to the ability of technology to support a task [197]. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

A system may be usable for some tasks and less usable for others; it may be usable for some users but not for others.

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Users actively repurpose tools to make them more personally usable and relevant. Design should support such repurposing. For example, Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use. Users who explore available commands in a tool are better at repurposing its functionality. More surprisingly, engaging in technical reasoning (reasoning about functionality and objects) supports repurposing more than procedural knowledge inherited from other software.

Tversky and Jamalian [833] proposed that embodied action is at the core of this. We move our bodies and toss, push, and pull objects. These movements can be thought about, imagined, and referred to in language. This, in turn, can change the substrate of thinking.

Davis [180] proposed that whether an individual ends up using a system, that is, their usage behavior, depends on their intention to use the system.

The theory of task–technology fit (TTF) can illuminate what users consider useful and how this affects their decision to adopt a particular technology. TTF refers to the ability of technology to support a task [197]. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect. TTF theory posits that a rational user will choose the tool with the highest fit due to its efficacy and efficiency. Conversely, a system that does not offer a good fit will not be used.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it [180]. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use.

Students who learned to do calculations with an abacus solve mathematical problems differently from others [796]. They rely more on mental imagery of the movement of beads on the abacus, which makes their mental calculations highly efficient for certain types of calculations.

For example, augmentative and alternative communication (AAC) is concerned with supporting non-speaking individuals with motor disabilities. AAC users rely on speech-generating devices (SGDs) to communicate with other people.

TTF has been used to assess users' willingness to use various technologies such as email or spreadsheets.

They provided an example of the usability of software installation. This was quantified through the time it takes to install software.

For example, a scrollbar is an interaction instrument, or tool, that operates on documents.

a user using a system to accomplish a task is not markedly different from a person using a hammer to drive nails or an algebraic rule to do calculations in one's head.

While a tool can enhance performance in cognitively challenging tasks, its extended use may erode the cognitive capability of the user.

The tool itself may become 'transparent' and we start perceiving 'through it.'

Using a tool for extended periods can fundamentally change the way a user thinks and perceives both the tool and the world.

accessibility concerns the match between a user's abilities and the system's required abilities.

TTF theory posits that a rational user will choose the tool with the highest fit due to its efficacy and efficiency. Conversely, a system that does not offer a good fit will not be used.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

usability is multidimensional. This means that in most settings, a valid characterization of usability will need to employ several dimensions and measures.

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Users actively repurpose tools to make them more personally usable and relevant.

Utility centers what users want from technology.

Usability is one of the best predictors of users' willingness to adopt software.

Cognitive integration means that we internalize the operation of the tool. We not only act but also start thinking as defined by the unique constraints and mechanisms of the tool.

accessibility concerns the match between a user's abilities and the system's required abilities. As such, it differs from usability (which is about the relationship between users, tools, and tasks) and utility (which is about whether a tool may be used to complete a task).

TTF refers to the ability of technology to support a task. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

The second dimension, social acceptability, concerns whether interactions map well to the social norms and roles in the settings where they occur.

Acceptability has two main dimensions. The first dimension, practical acceptability, includes costs, the reliability of the interactive system, and its compatibility with other systems. The perceptions of utility and usability may also influence the judgment of practical acceptability.

usability is multidimensional. This means that in most settings, a valid characterization of usability will need to employ several dimensions and measures.

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user.

Eye-typing is an effective means of communication; however, it is not efficient. Three fundamental problems prevent high entry rates. First, the eyes are sensory organs and not control organs. It is difficult for users to artificially maintain fixation on specific keys. Second, the dwell timeout provides a low ceiling on performance. Third, people think in terms of words, phrases, and sentences when they communicate. Eye-typing forces users to think in terms of individual letters.

Supple++ [266] is a computational method developed in HCI that can improve graphical user interfaces to better fit a user's unique motor and vision abilities. In Supple++, the user is first asked to perform a series of motor tasks. This information is used to calibrate an internal computational model of the user's motor ability. Once the calibration is complete, Supple++ optimizes the user interface automatically by changing the size and location of user interface elements and the organization of the user interface, subject to constraints specified by the designer.

Galletta et al. [267] warned against the effect of spell checkers on verbal ability. Having a spell checker in a word processing program may make users overly rely on the tool even if it makes several mistakes, both false positives and false negatives. The authors showed experimentally that university students who had a spell checker on during a document editing task had more errors left in the document than those who did not.

Blind cane users are a good example [756]. When blind users learn to sense the environment with a cane, their perception of tactile and auditory stimuli slowly changes. Instead of sensing stimuli close to their hand, when they hold the cane, they can integrate tactile (vibration) and auditory stimuli close to the tip of the cane. They develop multimodal, integrated percepts that correspond to the tip of the cane.

Students who learned to do calculations with an abacus solve mathematical problems differently from others [796]. They rely more on mental imagery of the movement of beads on the abacus, which makes their mental calculations highly efficient for certain types of calculations.

Beaudouin-Lafon [53] departed from the idea that the manipulation of physical objects with our hands can be used as the basis for designing new user interfaces. He separated domain objects that are manipulated from interaction instruments, which are computer artifacts that manipulate domain objects. For example, a scrollbar is an interaction instrument, or tool, that operates on documents. Further analysis reveals it has low integration because a 1D action is controlled by a 2D mouse, and it has low compatibility in some designs because the content moves in a different direction from the movement of the scrollbar.

Research has drawn from linguistics, especially pragmatics, to understand how the way we talk with computers changes depending on the communication context.

According to Suchman, robustness is a key consideration in the design of dialogue. Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

HCI researchers have developed a rich palette of theories to understand such dialogues. These theories explain what happens in dialogue and how it shapes the relationship between the partners. These theories also have implications for how we design interaction.

Comparing mode-based interactions. A device is designed to allow users to control the relative humidity in their house. The device has two modes. In Automatic mode, the system keeps the relative humidity in the 50%–60% range. In the Manual mode, the user can set the desired level of relative humidity and the system will attempt to maintain it. The device is a small wall-mounted unit with the following UI elements. (a) The visual display indicates the current level of relative humidity and whether the system is in Automatic or Manual mode. (b) The "–" and "+" buttons enable the user to reduce or increase the desired level of relative humidity, respectively. (c) The "Automatic" button puts the system in Automatic mode. If the user pushes the "–" or "+" button, the system switches to Manual mode and remains in that mode until the user pushes the "Automatic" button. (a) Draw a state diagram for this system. (b) By viewing interaction with this system as goal-directed action, explain the steps comprising the gulf of evaluation and the gulf of execution for this UI. (c) State the type and level of automation of this system. (d) Is this system a mixed-initiative interface? Justify your answer.

Mixed-initiative interfaces. Pick any AI-assisted feature that you are familiar with. Assess it against Horvitz's principles of mixed-initiative interfaces.

Gulfs. Pick a graphical user interface, for example, something you use for education. Then, choose a task, for example, "sending a message to the teacher." Assess this task through the lens of Norman's two gulfs: the gulf of evaluation and the gulf of execution.

Theories of human–computer dialogue. Consider the following potential dialogue interfaces: (a) a user interacting with an automated chat agent from an airline to resolve a delayed flight; (b) a child uploading homework using a web interface; and (c) a user who is trying to show a picture on their mobile phone on a nearby television screen. Make any necessary assumptions about the interfaces and discuss which model of dialogue would provide the most insight for each interface: (a) FSMs, (b) dialogue as goal-directed action, (c) dialogue as embodied action, or (d) dialogue from a communication perspective.

Communication partners: Who are the actors in the dialogue? Communication goals: What is the final state the computer should be in for the user to consider the task completed? Communication act: What are the possible communication acts? In other words, what are the possible utterances or messages that can be delivered? Communication sequence: Draw a sequence of the communication turns leading to the goal, similar to Figure 18.1. Initiative: To which degree can each partner initiate communication on their own? Cue: Which cues are shown to help the user understand the state of the computer? Feedback: Which cues are shown to help the user understand the effects of their communication acts?

Core concepts of dialogue interaction. Dialogue offers a rich conceptual framework for understanding interaction. First, choose an everyday interaction with which you are familiar. It can be anything from filling out a form to chatting with a chatbot. Then, choose a particular dialogue to focus on, for example, creating a user account or printing a document. Now, provide the following information for the dialogue:

Generally, it is beneficial when mixed-initiative interfaces learn and adapt to individual users.

Because users' goals and situations change over time, the system is never "ready."

The feasible communication acts and their effects are conditioned by the state of the partner.

The paradoxical effect of hyperarticulation is that despite trying to improve understanding, it can make speech recognition worse.

When an automated action is taken, it is important to consider the timing, as incorrectly timed automated actions can distract the user.

If there is ambiguity about what the user wants and wrong automation might harm the user, the system should ask for more information or not carry out the command.

Since the system will be unlikely to always automate functions successfully, it is important that users can directly trigger and terminate functions.

If the system is uncertain about the user's intent, the system should ask the user after having considered the cost of interrupting the user.

If a system operates under a high uncertainty of the user's goals, the system should perform less automation to avoid interrupting the user with poor suggestions.

When there is a misunderstanding about the context of the dialogue, errors may happen, and the partners must recover from them.

If the supervising user wants to intervene, the gulf of evaluation becomes relevant.

The mapping requires the user to figure out how to accomplish a goal with an interface. It implies that "The user must translate the psychological goals and intentions into the desired system state, then determine what settings of the control mechanisms will yield that state, and then determine what physical manipulations of the mechanism are required" [600, p. 37].

In direct manipulation interfaces (Chapter 28), the visual presentation of an object resembles its physical correspondent and can be directly acted on. For example, text in a text editor can be highlighted, deleted, or changed by point-and-click-style interactions [416].

the seven-stage model of interaction proposed by Norman [600] applies to all modalities of interaction

They also aid design and engineering by highlighting desirable properties of a dialogue system [5].

This AAC system was designed to use context to facilitate the creation of personal narratives [75].

The cognitive scientist Kirsh presented a criticism of Norman's view of dialogue and developed an alternative based on the theory of embodied cognition [416].

Horvitz [360] summarized the principles of mixed-initiative interfaces as follows:

A research group at the University of Washington [60] recruited 10 families and recorded their communications with Amazon Echo Dot (Alexa) for four weeks.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other [60].

Section 18.3 outlines a view of dialogue developed by Suchman [804] that emphasizes the situated nature of dialogue.

A cornerstone of this research is the book Plans and Situated Action: The Problem of Human–Machine Communication by Suchman [804].

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot:

Norman's model stresses the need for users' acts to be understood by the computer and for users to understand the computer. Successful interfaces should also "provide a strong sense of understanding and control" [600, p. 49].

Affordance, which we discussed in Chapter 3, refers to how well users can interpret what actions are possible with a widget. Visibility is a handy related concept in design that underlies direct manipulation interfaces [416].

Norman suggested that the ease of mapping is related to its directness, "where directness can be measured by the complexity of the relationship between representation and value, measured by the length of the description of the mapping" [600, pp. 28–29].

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation (Figure 18.2).

A significant early theory of dialogue interaction is the seven-stage model of Norman [600]. It considers interaction as goal-directed, turn-based dialogue.

Liu and Chilton [488] studied prompt engineering for text-to-image generation; see the figure in this paper example box, which shows examples of answers to the prompt "SUBJECT in the style of STYLE."

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must "engage in bruteforce trial and error with the text prompt when the result quality is poor."

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option. In practice, we do not always know what the options mean or even what options are available. Kirsh argued that users need to actively explore interfaces to become aware of the available functions and how they work.

Code-switching refers to a switch in language to match the capabilities of the communication partner. For example, you likely use different language when talking with friends and with family.

Human–machine interaction, according to Suchman, is similar to but different from human–human dialogue. It is similar in the sense that people pursue a shared understanding: They actively work to make themselves understood. It is different in the sense that the communication abilities of computers are limited, which requires humans to adapt.

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation. These two concepts describe inferential breakpoints for users seeking to express their intentions and interpret feedback from the system, respectively.

The model subscribes to a theoretical assumption about dialogue: The defining cognitive challenge in dialogue is understanding the communication partner such that the appropriate next turn can be taken. In other words, the dialogue is intentional or goal-directed: Users aim to drive the computer to a particular desired state.

A significant early theory of dialogue interaction is the seven-stage model of Norman [600]. It considers interaction as goal-directed, turn-based dialogue.

The cognitive scientist Kirsh presented a criticism of Norman's view of dialogue and developed an alternative based on the theory of embodied cognition [416].

The How was School today. . .? concept was developed iteratively with two children with CP and the help of school staff. This AAC system was designed to use context to facilitate the creation of personal narratives [75].

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot:

Horvitz [360] summarized the principles of mixed-initiative interfaces as follows:

A research group at the University of Washington [60] recruited 10 families and recorded their communications with Amazon Echo Dot (Alexa) for four weeks.

According to Suchman, robustness is a key consideration in the design of dialogue.

A cornerstone of this research is the book Plans and Situated Action: The Problem of Human–Machine Communication by Suchman [804].

Liu and Chilton [488] studied prompt engineering for text-to-image generation

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must

Gulf of evaluation: This gulf refers to knowing how a perceived change in the computer has moved it closer to the intended goal state.

Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.