Users actively repurpose tools to make them more personally usable and relevant. Design should support such repurposing. For example, Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use. Users who explore available commands in a tool are better at repurposing its functionality. More surprisingly, engaging in technical reasoning (reasoning about functionality and objects) supports repurposing more than procedural knowledge inherited from other software.

Tversky and Jamalian [833] proposed that embodied action is at the core of this. We move our bodies and toss, push, and pull objects. These movements can be thought about, imagined, and referred to in language. This, in turn, can change the substrate of thinking.

Davis [180] proposed that whether an individual ends up using a system, that is, their usage behavior, depends on their intention to use the system.

The theory of task–technology fit (TTF) can illuminate what users consider useful and how this affects their decision to adopt a particular technology. TTF refers to the ability of technology to support a task [197]. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect. TTF theory posits that a rational user will choose the tool with the highest fit due to its efficacy and efficiency. Conversely, a system that does not offer a good fit will not be used.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it [180]. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use.

Students who learned to do calculations with an abacus solve mathematical problems differently from others [796]. They rely more on mental imagery of the movement of beads on the abacus, which makes their mental calculations highly efficient for certain types of calculations.

For example, augmentative and alternative communication (AAC) is concerned with supporting non-speaking individuals with motor disabilities. AAC users rely on speech-generating devices (SGDs) to communicate with other people.

TTF has been used to assess users' willingness to use various technologies such as email or spreadsheets.

They provided an example of the usability of software installation. This was quantified through the time it takes to install software.

For example, a scrollbar is an interaction instrument, or tool, that operates on documents.

a user using a system to accomplish a task is not markedly different from a person using a hammer to drive nails or an algebraic rule to do calculations in one's head.

While a tool can enhance performance in cognitively challenging tasks, its extended use may erode the cognitive capability of the user.

The tool itself may become 'transparent' and we start perceiving 'through it.'

Using a tool for extended periods can fundamentally change the way a user thinks and perceives both the tool and the world.

accessibility concerns the match between a user's abilities and the system's required abilities.

TTF theory posits that a rational user will choose the tool with the highest fit due to its efficacy and efficiency. Conversely, a system that does not offer a good fit will not be used.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

usability is multidimensional. This means that in most settings, a valid characterization of usability will need to employ several dimensions and measures.

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Users actively repurpose tools to make them more personally usable and relevant.

Utility centers what users want from technology.

Usability is one of the best predictors of users' willingness to adopt software.

Cognitive integration means that we internalize the operation of the tool. We not only act but also start thinking as defined by the unique constraints and mechanisms of the tool.

accessibility concerns the match between a user's abilities and the system's required abilities. As such, it differs from usability (which is about the relationship between users, tools, and tasks) and utility (which is about whether a tool may be used to complete a task).

TTF refers to the ability of technology to support a task. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

The second dimension, social acceptability, concerns whether interactions map well to the social norms and roles in the settings where they occur.

Acceptability has two main dimensions. The first dimension, practical acceptability, includes costs, the reliability of the interactive system, and its compatibility with other systems. The perceptions of utility and usability may also influence the judgment of practical acceptability.

usability is multidimensional. This means that in most settings, a valid characterization of usability will need to employ several dimensions and measures.

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user.

Eye-typing is an effective means of communication; however, it is not efficient. Three fundamental problems prevent high entry rates. First, the eyes are sensory organs and not control organs. It is difficult for users to artificially maintain fixation on specific keys. Second, the dwell timeout provides a low ceiling on performance. Third, people think in terms of words, phrases, and sentences when they communicate. Eye-typing forces users to think in terms of individual letters.

Supple++ [266] is a computational method developed in HCI that can improve graphical user interfaces to better fit a user's unique motor and vision abilities. In Supple++, the user is first asked to perform a series of motor tasks. This information is used to calibrate an internal computational model of the user's motor ability. Once the calibration is complete, Supple++ optimizes the user interface automatically by changing the size and location of user interface elements and the organization of the user interface, subject to constraints specified by the designer.

Galletta et al. [267] warned against the effect of spell checkers on verbal ability. Having a spell checker in a word processing program may make users overly rely on the tool even if it makes several mistakes, both false positives and false negatives. The authors showed experimentally that university students who had a spell checker on during a document editing task had more errors left in the document than those who did not.

Blind cane users are a good example [756]. When blind users learn to sense the environment with a cane, their perception of tactile and auditory stimuli slowly changes. Instead of sensing stimuli close to their hand, when they hold the cane, they can integrate tactile (vibration) and auditory stimuli close to the tip of the cane. They develop multimodal, integrated percepts that correspond to the tip of the cane.

Students who learned to do calculations with an abacus solve mathematical problems differently from others [796]. They rely more on mental imagery of the movement of beads on the abacus, which makes their mental calculations highly efficient for certain types of calculations.

Beaudouin-Lafon [53] departed from the idea that the manipulation of physical objects with our hands can be used as the basis for designing new user interfaces. He separated domain objects that are manipulated from interaction instruments, which are computer artifacts that manipulate domain objects. For example, a scrollbar is an interaction instrument, or tool, that operates on documents. Further analysis reveals it has low integration because a 1D action is controlled by a 2D mouse, and it has low compatibility in some designs because the content moves in a different direction from the movement of the scrollbar.

a scrollbar is an interaction instrument, or tool, that operates on documents

Galletta et al. [267] warned against the effect of spell checkers on verbal ability. Having a spell checker in a word processing program may make users overly rely on the tool even if it makes several mistakes, both false positives and false negatives. The authors showed experimentally that university students who had a spell checker on during a document editing task had more errors left in the document than those who did not.

Blind cane users are a good example [756]. When blind users learn to sense the environment with a cane, their perception of tactile and auditory stimuli slowly changes. Instead of sensing stimuli close to their hand, when they hold the cane, they can integrate tactile (vibration) and auditory stimuli close to the tip of the cane.

For example, the abacus is a wooden device used for teaching basic calculations. It consists of a frame with rows of wires along which beads can slide. Students who learned to do calculations with an abacus solve mathematical problems differently from others [796]. They rely more on mental imagery of the movement of beads on the abacus, which makes their mental calculations highly efficient for certain types of calculations.

TTF has been used to assess users' willingness to use various technologies such as email or spreadsheets.

For example, social acceptability was an important consideration for early smart glasses, that is, eyewear with computational capabilities, particularly models fitted with cameras [426].

For instance, Nielsen and Levy [592] compared users' performance and their preferences across 57 studies and found what they called a strong positive correlation. Nevertheless, they concluded that "there are still many cases in which users prefer systems that are measurably worse for them, so one should exercise caution" [p. 75].

For instance, Whiteside et al. [886] showed how to make explicit quantitative goals for usability. They provided an example of the usability of software installation. This was quantified through the time it takes to install software. This could take one hour or, in the best case, just 10 minutes.

For example, Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use. Users who explore available commands in a tool are better at repurposing its functionality.

For example, Koelle et al. [426] studied the adoption of data glasses (e.g., Google Glass, Meta Pro) over multiple years. They asked experts familiar with data glasses what would need to be improved to make data glasses more acceptable. Usefulness, functionality, and usability were the most important factors—more important than security, privacy, pricing, experience, and compatibility.

For example, a scrollbar is an interaction instrument, or tool, that operates on documents. Further analysis reveals it has low integration because a 1D action is controlled by a 2D mouse, and it has low compatibility in some designs because the content moves in a different direction from the movement of the scrollbar.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it.

Cognitive integration means that we internalize the operation of the tool.

TTF refers to the ability of technology to support a task. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

acceptability of the tool, that is, whether users choose to use the tool when given that option.

One prominent definition of accessibility is given by ISO 9241-171, which defines it as

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

the idea of tool use in human–computer interaction (HCI) is that a computer system is a tool for controlling something else.

Cognitive integration means that we internalize the operation of the tool.

TTF refers to the ability of technology to support a task [197]. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

One prominent definition of accessibility is given by ISO 9241-171, which defines it as "the usability of a product, service, environment or facility by people with the widest range of capabilities."

The second dimension, social acceptability, concerns whether interactions map well to the social norms and roles in the settings where they occur.

The first dimension, practical acceptability, includes costs, the reliability of the interactive system, and its compatibility with other systems.

acceptability of the tool, that is, whether users choose to use the tool when given that option.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

the "extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

What makes these objects tools is that they are not attached to the body but can be held to bring about changes in the condition of other objects [772]. By extension, the idea of tool use in human–computer interaction (HCI) is that a computer system is a tool for controlling something else.

Cognitive integration means that we internalize the operation of the tool.

accessibility is given by ISO 9241-171, which defines it as 'the usability of a product, service, environment or facility by people with the widest range of capabilities.'

TTF refers to the ability of technology to support a task

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it

social acceptability, concerns whether interactions map well to the social norms and roles in the settings where they occur

practical acceptability, includes costs, the reliability of the interactive system, and its compatibility with other systems

acceptability of the tool, that is, whether users choose to use the tool when given that option

usability is multidimensional

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

the ISO 9241-11 definition, based on work by Bevan and many others, which defines usability as the 'extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.'

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task.

utility is 'whether the functionality of a system in principle can do what is needed'

The utility of an interactive system concerns its match with the tasks of users.

tools may be more or less easy to manipulate when used. This is called usability

understanding computers as tools emphasizes the utility of a tool, by which we mean how well it supports what people want to do

Newman and Taylor presented a concrete approach to selecting usability measures, which they called selecting critical parameters, that is,

Research has shown that SUS can discriminate between systems with poor and good usability, can be used with a range of technologies, correlates modestly with task performance, correlates well with other questionnaires, and has good reliability.

The UMUX-Lite consists of just two items,

The answers to these questions are summed to calculate the overall SUS score. For odd-numbered items, subtract 1 from each score (1–5); for even-numbered items, subtract each score from 5. Then, sum these values to obtain the total SUS score.

The SUS consists of 10 questions answered on a Likert scale (from

The User Burden Scale is a questionnaire for measuring the felt burden in software use. It consists of six subscales: difficulty of use, physical burden, time and social burden, mental and emotional burden, privacy burden, and financial burden.

Research has drawn from linguistics, especially pragmatics, to understand how the way we talk with computers changes depending on the communication context.

According to Suchman, robustness is a key consideration in the design of dialogue. Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

HCI researchers have developed a rich palette of theories to understand such dialogues. These theories explain what happens in dialogue and how it shapes the relationship between the partners. These theories also have implications for how we design interaction.

Comparing mode-based interactions. A device is designed to allow users to control the relative humidity in their house. The device has two modes. In Automatic mode, the system keeps the relative humidity in the 50%–60% range. In the Manual mode, the user can set the desired level of relative humidity and the system will attempt to maintain it. The device is a small wall-mounted unit with the following UI elements. (a) The visual display indicates the current level of relative humidity and whether the system is in Automatic or Manual mode. (b) The "–" and "+" buttons enable the user to reduce or increase the desired level of relative humidity, respectively. (c) The "Automatic" button puts the system in Automatic mode. If the user pushes the "–" or "+" button, the system switches to Manual mode and remains in that mode until the user pushes the "Automatic" button. (a) Draw a state diagram for this system. (b) By viewing interaction with this system as goal-directed action, explain the steps comprising the gulf of evaluation and the gulf of execution for this UI. (c) State the type and level of automation of this system. (d) Is this system a mixed-initiative interface? Justify your answer.

Mixed-initiative interfaces. Pick any AI-assisted feature that you are familiar with. Assess it against Horvitz's principles of mixed-initiative interfaces.

Gulfs. Pick a graphical user interface, for example, something you use for education. Then, choose a task, for example, "sending a message to the teacher." Assess this task through the lens of Norman's two gulfs: the gulf of evaluation and the gulf of execution.

Theories of human–computer dialogue. Consider the following potential dialogue interfaces: (a) a user interacting with an automated chat agent from an airline to resolve a delayed flight; (b) a child uploading homework using a web interface; and (c) a user who is trying to show a picture on their mobile phone on a nearby television screen. Make any necessary assumptions about the interfaces and discuss which model of dialogue would provide the most insight for each interface: (a) FSMs, (b) dialogue as goal-directed action, (c) dialogue as embodied action, or (d) dialogue from a communication perspective.

Communication partners: Who are the actors in the dialogue? Communication goals: What is the final state the computer should be in for the user to consider the task completed? Communication act: What are the possible communication acts? In other words, what are the possible utterances or messages that can be delivered? Communication sequence: Draw a sequence of the communication turns leading to the goal, similar to Figure 18.1. Initiative: To which degree can each partner initiate communication on their own? Cue: Which cues are shown to help the user understand the state of the computer? Feedback: Which cues are shown to help the user understand the effects of their communication acts?

Core concepts of dialogue interaction. Dialogue offers a rich conceptual framework for understanding interaction. First, choose an everyday interaction with which you are familiar. It can be anything from filling out a form to chatting with a chatbot. Then, choose a particular dialogue to focus on, for example, creating a user account or printing a document. Now, provide the following information for the dialogue:

Generally, it is beneficial when mixed-initiative interfaces learn and adapt to individual users.

Because users' goals and situations change over time, the system is never "ready."

The feasible communication acts and their effects are conditioned by the state of the partner.

The paradoxical effect of hyperarticulation is that despite trying to improve understanding, it can make speech recognition worse.

When an automated action is taken, it is important to consider the timing, as incorrectly timed automated actions can distract the user.

If there is ambiguity about what the user wants and wrong automation might harm the user, the system should ask for more information or not carry out the command.

Since the system will be unlikely to always automate functions successfully, it is important that users can directly trigger and terminate functions.

If the system is uncertain about the user's intent, the system should ask the user after having considered the cost of interrupting the user.

If a system operates under a high uncertainty of the user's goals, the system should perform less automation to avoid interrupting the user with poor suggestions.

When there is a misunderstanding about the context of the dialogue, errors may happen, and the partners must recover from them.

If the supervising user wants to intervene, the gulf of evaluation becomes relevant.

The mapping requires the user to figure out how to accomplish a goal with an interface. It implies that "The user must translate the psychological goals and intentions into the desired system state, then determine what settings of the control mechanisms will yield that state, and then determine what physical manipulations of the mechanism are required" [600, p. 37].

In direct manipulation interfaces (Chapter 28), the visual presentation of an object resembles its physical correspondent and can be directly acted on. For example, text in a text editor can be highlighted, deleted, or changed by point-and-click-style interactions [416].

the seven-stage model of interaction proposed by Norman [600] applies to all modalities of interaction

They also aid design and engineering by highlighting desirable properties of a dialogue system [5].

This AAC system was designed to use context to facilitate the creation of personal narratives [75].

The cognitive scientist Kirsh presented a criticism of Norman's view of dialogue and developed an alternative based on the theory of embodied cognition [416].

Horvitz [360] summarized the principles of mixed-initiative interfaces as follows:

A research group at the University of Washington [60] recruited 10 families and recorded their communications with Amazon Echo Dot (Alexa) for four weeks.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other [60].

Section 18.3 outlines a view of dialogue developed by Suchman [804] that emphasizes the situated nature of dialogue.

A cornerstone of this research is the book Plans and Situated Action: The Problem of Human–Machine Communication by Suchman [804].

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot:

Norman's model stresses the need for users' acts to be understood by the computer and for users to understand the computer. Successful interfaces should also "provide a strong sense of understanding and control" [600, p. 49].

Affordance, which we discussed in Chapter 3, refers to how well users can interpret what actions are possible with a widget. Visibility is a handy related concept in design that underlies direct manipulation interfaces [416].

Norman suggested that the ease of mapping is related to its directness, "where directness can be measured by the complexity of the relationship between representation and value, measured by the length of the description of the mapping" [600, pp. 28–29].

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation (Figure 18.2).

A significant early theory of dialogue interaction is the seven-stage model of Norman [600]. It considers interaction as goal-directed, turn-based dialogue.

Liu and Chilton [488] studied prompt engineering for text-to-image generation; see the figure in this paper example box, which shows examples of answers to the prompt "SUBJECT in the style of STYLE."

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must "engage in bruteforce trial and error with the text prompt when the result quality is poor."

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option. In practice, we do not always know what the options mean or even what options are available. Kirsh argued that users need to actively explore interfaces to become aware of the available functions and how they work.

Code-switching refers to a switch in language to match the capabilities of the communication partner. For example, you likely use different language when talking with friends and with family.

Human–machine interaction, according to Suchman, is similar to but different from human–human dialogue. It is similar in the sense that people pursue a shared understanding: They actively work to make themselves understood. It is different in the sense that the communication abilities of computers are limited, which requires humans to adapt.

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation. These two concepts describe inferential breakpoints for users seeking to express their intentions and interpret feedback from the system, respectively.

The model subscribes to a theoretical assumption about dialogue: The defining cognitive challenge in dialogue is understanding the communication partner such that the appropriate next turn can be taken. In other words, the dialogue is intentional or goal-directed: Users aim to drive the computer to a particular desired state.

A significant early theory of dialogue interaction is the seven-stage model of Norman [600]. It considers interaction as goal-directed, turn-based dialogue.

The cognitive scientist Kirsh presented a criticism of Norman's view of dialogue and developed an alternative based on the theory of embodied cognition [416].

The How was School today. . .? concept was developed iteratively with two children with CP and the help of school staff. This AAC system was designed to use context to facilitate the creation of personal narratives [75].

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot:

Horvitz [360] summarized the principles of mixed-initiative interfaces as follows:

A research group at the University of Washington [60] recruited 10 families and recorded their communications with Amazon Echo Dot (Alexa) for four weeks.

According to Suchman, robustness is a key consideration in the design of dialogue.

A cornerstone of this research is the book Plans and Situated Action: The Problem of Human–Machine Communication by Suchman [804].

Liu and Chilton [488] studied prompt engineering for text-to-image generation

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must

Gulf of evaluation: This gulf refers to knowing how a perceived change in the computer has moved it closer to the intended goal state.

Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

Kirsh proposed an alternate model, showing that every stage in Norman's model can have an interactive relationship with the environment. We learn about options by exploring the interface, discover how to specify actions by trying them out and observing the outcomes, position our bodies to better perceive environmental responses, and adjust the environment to facilitate response evaluation.

The cognitive scientist Kirsh presented a criticism of Norman's view of dialogue and developed an alternative based on the theory of embodied cognition [416].

Horvitz [360] summarized the principles of mixed-initiative interfaces as follows:

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other [60].

A cornerstone of this research is the book Plans and Situated Action: The Problem of Human–Machine Communication by Suchman [804].

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation (Figure 18.2). These two concepts describe inferential breakpoints for users seeking to express their intentions and interpret feedback from the system, respectively.

The How was School today. . .? concept was developed iteratively with two children with CP and the help of school staff. This AAC system was designed to use context to facilitate the creation of personal narratives [75].

The cognitive scientist Kirsh presented a criticism of Norman's view of dialogue and developed an alternative based on the theory of embodied cognition [416].

Horvitz [360] summarized the principles of mixed-initiative interfaces as follows:

A research group at the University of Washington [60] recruited 10 families and recorded their communications with Amazon Echo Dot (Alexa) for four weeks.

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot.

Liu and Chilton [488] studied prompt engineering for text-to-image generation.

Liu and Chilton [488] noted that interaction with such models faces a dilemma.

Section 18.3 outlines a view of dialogue developed by Suchman [804] that emphasizes the situated nature of dialogue.

A cornerstone of this research is the book Plans and Situated Action: The Problem of Human–Machine Communication by Suchman [804].

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation (Figure 18.2).

A significant early theory of dialogue interaction is the seven-stage model of Norman [600].

FSMs, as formal accounts of dialogue, are limited to transitions in a dialogue. They do not make assumptions about the way options or feedback are presented to the user. The same FSM could be implemented as an interface in multiple ways. FSMs do not make explicit assumptions about the user, either: FSMs are mute about how users perceive, reason, learn, and experience.

Formal models of computation are suitable for describing discrete, moded dialogues. A mode refers to the variation in the interpretation of a user's input according to an internal state.

The model subscribes to a theoretical assumption about dialogue: The defining cognitive challenge in dialogue is understanding the communication partner such that the appropriate next turn can be taken. In other words, the dialogue is intentional or goal-directed: Users aim to drive the computer to a particular desired state.

Kirsh argued that we are not just passively reacting to computer-generated options. If we look at interaction at a higher level, beyond a single action, we see that users are also actively influencing their environments.

It is different in the sense that the communication abilities of computers are limited, which requires humans to adapt.

users adapt their approaches to match the capabilities of the machine rather than the other way around.

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option. In practice, we do not always know what the options mean or even what options are available.

One thing that is missing is an account of how beliefs about the computer are formed and updated and how they drive action specification. The current understanding is that users form internal models that predict how their actions produce perceived outputs, and they learn to minimize prediction errors. This explains why people explore interfaces (to develop better internal models) and why, eventually, they no longer need to compare outcomes against goals.

Norman's model stresses the need for users' acts to be understood by the computer and for users to understand the computer. Successful interfaces should also "provide a strong sense of understanding and control"

The current understanding is that users form internal models that predict how their actions produce perceived outputs, and they learn to minimize prediction errors. This explains why people explore interfaces (to develop better internal models) and why, eventually, they no longer need to compare outcomes against goals.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative. Unlike in the case of an FSM, the computing system can take action without a command from the user; the initiative is mixed.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

Formally, an FSM is a tuple (Σ, S,s0, δ, F), where: Σ is the input, that is, a finite set of symbols; S is a finite set of states or modes; s0 ∈ S is the initial state; δ is the state transition function δ : S × Σ→S; F is the set of final states, that is, a subset of S.

Gulf of evaluation: This gulf refers to knowing how a perceived change in the computer has moved it closer to the intended goal state. For example, imagine setting the temperature of an intelligent thermostat and not perceiving an immediate effect. How can you tell if your command had the desired effect on the system?

Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer. For example, what should you do to get a piece of text copied to the clipboard and pasted in a specific location?

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context. The act aims to get the other partner to do or understand something. This understanding then forms the context within which the other partner takes their turn.

Users are "architects" of their environments, as Kirsh put it. For example, users may change the settings to turn on or off a function or change the way it behaves. They also choose the applications they use. Such tailoring behaviors are not explained by Norman's intention–action–response–interpretation–evaluation cycle.

The How was School today. . .? concept was developed iteratively with two children with CP and the help of school staff. This AAC system was designed to use context to facilitate the creation of personal narratives [75]. The authors called their approach "data-to-text": The idea was to add sensors to the environment and the wheelchair of a user.

The families also exaggerated articulation, a phenomenon known as hyperarticulation. The paradoxical effect of hyperarticulation is that despite trying to improve understanding, it can make speech recognition worse.

Their study exposed the limited nature of contemporary speech interaction from a conversational perspective. Although breakdowns were not that frequent—one occurred every four hours of use—they disrupted regular use and often required joint effort to overcome.

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot: Supervisor, Operator, Peer, Bystander.

The results of the studies show that a small set of responses (3–9) may be sufficient to generate an idea of what a prompt can do; the computation of more responses might just waste time. The results also show that the SUBJECT would sometimes get lost in the STYLE; some prompts inadvertently led to grotesque or inappropriate images.

The challenge here is sometimes described as prompt engineering—the search for prompts that give the output the user finds adequate for the task.

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must "engage in bruteforce trial and error with the text prompt when the result quality is poor."

A good design allows users to try out options without penalizing trials.