Their study exposed the limited nature of contemporary speech interaction from a conversational perspective. Although breakdowns were not that frequent—one occurred every four hours of use—they disrupted regular use and often required joint effort to overcome.

According to Scholtz [745], the two gulfs manifest differently in the different roles a user may have when interacting with a robot: Supervisor, Operator, Peer, Bystander.

The results of the studies show that a small set of responses (3–9) may be sufficient to generate an idea of what a prompt can do; the computation of more responses might just waste time. The results also show that the SUBJECT would sometimes get lost in the STYLE; some prompts inadvertently led to grotesque or inappropriate images.

The challenge here is sometimes described as prompt engineering—the search for prompts that give the output the user finds adequate for the task.

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must "engage in bruteforce trial and error with the text prompt when the result quality is poor."

A good design allows users to try out options without penalizing trials.

Kirsh argued that we are not just passively reacting to computer-generated options.

Kirsh argued that users need to actively explore interfaces to become aware of the available functions and how they work.

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option.

Generally, it is beneficial when mixed-initiative interfaces learn and adapt to individual users.

A mixed-initiative interface needs to infer the user's goals so that it can act upon them.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

According to Suchman, robustness is a key consideration in the design of dialogue.

Human–machine interaction, according to Suchman, is similar to but different from human–human dialogue.

Communication between humans and computers is a long-standing topic of HCI research.

FSMs are an effective way to describe how dialogue is structured; however, they are limited to memory-free dialogue.

An FSM can be graphically presented as a graph where nodes represent states and edges represent transitions.

An FSM is a model of discrete computation applicable to dialogues.

A mode refers to the variation in the interpretation of a user's input according to an internal state.

Dialogue can be described using models of computation from computer science.

Affordance, which we discussed in Chapter 3, refers to how well users can interpret what actions are possible with a widget.

Mapping and feedback are crucial concepts for understanding a computer's turn.

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation.

A significant early theory of dialogue interaction is the seven-stage model of Norman [600].

Fourth, both the computer and the user may have initiative.

Third, in dialogue, both the computer and the human participate in establishing a shared context.

Second, both dialogues in Figure 18.1 evolve in a turn-based manner.

Both are forms of dialogue.

This broad definition has several immediate and important consequences for HCI.

In one turn, an appropriate communication act is made by one partner based on the communication context.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns.

The dialogue view of interaction evolved early in the history of computing.

In human–computer interaction (HCI), the communication partners are an interactive system and the user of such a system.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

Formal models of computation are suitable for describing discrete, moded dialogues. A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

prompt engineering—the search for prompts that give the output the user finds adequate for the task.

Visibility is a handy related concept in design that underlies direct manipulation interfaces.

Affordance, which we discussed in Chapter 3, refers to how well users can interpret what actions are possible with a widget.

Mapping requires the user to figure out how to accomplish a goal with an interface.

An FSM is a model of discrete computation applicable to dialogues. In computer science, an FSM is a special case of a Turing machine that reads but does not write on the tape.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other.

Users are 'architects' of their environments, as Kirsh put it.

Kirsh argued that the discoverability of such options is as important as their visibility; however, discoverability is not covered well by Norman's theory.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other.

Employing dialogue to resolve key uncertainties: If the system is uncertain about the user's intent, the system should ask the user after having considered the cost of interrupting the user. While ambiguities can be resolved via dialogue, this principle warns against always asking the user: Every interaction bears a cost (e.g., time and effort) that should be factored in when deciding whether and when to engage in dialogue.

The paradoxical effect of hyperarticulation is that despite trying to improve understanding, it can make speech recognition worse.

Users are 'architects' of their environments, as Kirsh put it. For example, users may change the settings to turn on or off a function or change the way it behaves. They also choose the applications they use. Such tailoring behaviors are not explained by Norman's intention–action–response–interpretation–evaluation cycle.

Fourth, both the computer and the user may have initiative. For example, a pop-up window can be presented to confirm a risky selection.

Therefore, the design of feedback, affordances, and cues is central to dialogue-based interaction.

Third, in dialogue, both the computer and the human participate in establishing a shared context. The computer does not simply receive a message; it also communicates the effects of that message.

Context, as we will see, can be understood through the concept of a state: The feasible communication acts and their effects are conditioned by the state of the partner.

In other words, in dialogue, acts of communication are conditional: The meaning of a turn depends on the communication context.

Siri's reply and the selection of "compose" would make no sense without the context provided by the preceding exchange.

Second, both dialogues in Figure 18.1 evolve in a turn-based manner. Each turn redefines the communication context.

For instance, the seven-stage model of interaction proposed by Norman [600] applies to all modalities of interaction; Section 18.1 explains this model in detail.

In Figure 18.1, two nominally different types of interaction are shown: speech-based and graphical. Both are forms of dialogue. This means that the concepts of dialogue are applicable across modalities.

This broad definition has several immediate and important consequences for HCI. First, dialogue, as a form of interaction, is not limited to speech and language even though this is often our first interpretation of the term "dialogue."

Initially, text consoles were used to provide access to programs running on mainframe computers.

This made it possible to structure interaction in the form of a dialogue, freeing the user from having to memorize commands, as required in command-based interfaces.

Kirsh argued that we are not just passively reacting to computer-generated options. If we look at interaction at a higher level, beyond a single action, we see that users are also actively influencing their environments. Users are "architects" of their environments, as Kirsh put it. For example, users may change the settings to turn on or off a function or change the way it behaves. They also choose the applications they use. Such tailoring behaviors are not explained by Norman's intention–action–response–interpretation–evaluation cycle.

From the 1960s to the 1980s, rapid advances in computer displays made it possible to present much richer information to users.

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option. In practice, we do not always know what the options mean or even what options are available. Kirsh argued that users need to actively explore interfaces to become aware of the available functions and how they work. Via exploration, they also learn about their own abilities in using them. Consider the first time you launch an application; you probably try out various actions to see what happens. Kirsh argued that the discoverability of such options is as important as their visibility; however, discoverability is not covered well by Norman's theory.

The modelling subscribes to a linear account of the cognitive mechanism, going from goals to actions and back. However, according to current understanding in cognitive sciences, the picture is more complicated. One thing that is missing is an account of how beliefs about the computer are formed and updated and how they drive action specification. The current understanding is that users form internal models that predict how their actions produce perceived outputs, and they learn to minimize prediction errors. This explains why people explore interfaces (to develop better internal models) and why, eventually, they no longer need to compare outcomes against goals. Moreover, the model was initially used in a weak, heuristic sense and did not converge with efforts to implement interactive systems.

The dialogue view of interaction evolved early in the history of computing.

These theories also have implications for how we design interaction.

These theories explain what happens in dialogue and how it shapes the relationship between the partners.

HCI researchers have developed a rich palette of theories to understand such dialogues.

In human–computer interaction (HCI), the communication partners are an interactive system and the user of such a system.

This view draws on what we know about human–human communication and, in particular, how conversations are structured; see Chapter 9 for a summary.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

One thing that is missing is an account of how beliefs about the computer are formed and updated and how they drive action specification. The current understanding is that users form internal models that predict how their actions produce perceived outputs, and they learn to minimize prediction errors.

Dialogue can be understood as computation, goal-directed action, communication, or embodied action.

Dialogue interaction includes speech-based and graphical interactions.

The core elements of dialogue are communication turns, the communication context, and turn interpretation.

Dialogue is about the organization of communication as a series of turns between communication partners.

A mode refers to the variation in the interpretation of a user's input according to an internal state.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other.

Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative.

An FSM is a model of discrete computation applicable to dialogues.

Dialogue can be described using models of computation from computer science.

Mapping and feedback are crucial concepts for understanding a computer's turn.

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation.

A significant early theory of dialogue interaction is the seven-stage model of Norman.

This understanding then forms the context within which the other partner takes their turn.

The act aims to get the other partner to do or understand something.

In one turn, an appropriate communication act is made by one partner based on the communication context.

Dialogue evolves through communication turns between two or more partners.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

both the computer and the human participate in establishing a shared context. The computer does not simply receive a message; it also communicates the effects of that message.

Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

Communication repair refers to the "work of restoring shared understanding" when conversational partners misunderstand each other.

Employing socially appropriate behaviors for agent–user interaction: Any interruptions by a system should be compatible with the social expectations of the user being interrupted and offered automated services. For example, social media feeds may integrate AI-generated and human-generated content without disclosing the source.

A mixed-initiative interface needs to infer the user's goals so that it can act upon them. Several machine learning techniques are potentially suitable depending on the specific AI problem.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative. Unlike in the case of an FSM, the computing system can take action without a command from the user; the initiative is mixed.

Liu and Chilton [488] studied prompt engineering for text-to-image generation

Liu and Chilton [488] noted that interaction with such models faces a dilemma. While it is possible to input anything as a prompt to such models, users must "engage in bruteforce trial and error with the text prompt when the result quality is poor." The challenge here is sometimes described as prompt engineering—the search for prompts that give the output the user finds adequate for the task.

New ways of interacting that rely on dialogue keep emerging; at the time of writing this book (early 2020s), large language models such as ChatGPT and Google Bard are making the headlines daily. The interaction with such models is primarily done through text prompts to which the model replies.

In interaction with interactive systems based on artificial intelligence, mixed-initiative dialogue is crucial. We discuss mixed-initiative interfaces in Section 18.4.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative.

Kirsh argued that we are not just passively reacting to computer-generated options. If we look at interaction at a higher level, beyond a single action, we see that users are also actively influencing their environments. Users are "architects" of their environments, as Kirsh put it.

Code-switching refers to a switch in language to match the capabilities of the communication partner... Such differences are important because depending on the communication context, people will have different expectations and styles they use in dialogue with a computer.

Consistency: Are the same actions available, and do they have the same consequences across similar states? Dialogue length: How many turns are needed to get from the initial state to the end state? Number of choices: The number of options available to the user is a predictor of choice reaction time. Error recovery cost: If an error is made, how many turns are needed to recover from it? Connectedness: Can final states be reached from all initial states? Strong connectedness: Can final states be reached from all initial states via a particular action? Reversibility: Can the effect of a given action be reversed in one action?

Dialogue can be described using models of computation from computer science. Such models include finite state machines (FSMs), pushdown automata, and Petri nets. These models can be expressed with formal languages, including context-free grammars and graphs, and they can be implemented in event handlers in user interface (UI) software.

An FSM is a tuple (Σ, S,s0, δ, F), where: • Σ is the input, that is, a finite set of symbols; • S is a finite set of states or modes; • s0 ∈ S is the initial state; • δ is the state transition function δ : S × Σ→S; • F is the set of final states, that is, a subset of S.

Gulf of evaluation: This gulf refers to knowing how a perceived change in the computer has moved it closer to the intended goal state.

Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

An FSM is a model of discrete computation applicable to dialogues. In computer science, an FSM is a special case of a Turing machine that reads but does not write on the tape.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context. The act aims to get the other partner to do or understand something. This understanding then forms the context within which the other partner takes their turn.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

Dialogue is about the organization of communication as a series of turns between communication partners. The core elements of dialogue are communication turns, the communication context, and turn interpretation. Dialogue interaction includes speech-based and graphical interactions. Dialogue can be understood as computation, goal-directed action, communication, or embodied action.

Kirsh argued that we are not just passively reacting to computer-generated options. If we look at interaction at a higher level, beyond a single action, we see that users are also actively influencing their environments. Users are 'architects' of their environments, as Kirsh put it.

Kirsh points out that Norman's model makes an unrealistic assumption: The user is assumed to know the environment and its options and is merely picking an option. In practice, we do not always know what the options mean or even what options are available. Kirsh argued that users need to actively explore interfaces to become aware of the available functions and how they work.

Mixed-initiative interaction is the idea of organizing interaction in dialogue where both the computer and the human can take initiative. Unlike in the case of an FSM, the computing system can take action without a command from the user; the initiative is mixed.

Code-switching refers to a switch in language to match the capabilities of the communication partner.

Communication repair refers to the 'work of restoring shared understanding' when conversational partners misunderstand each other.

Robustness refers to the communication partners' ability to achieve shared understanding even in light of misunderstandings and other unanticipated troubles.

Human–machine interaction, according to Suchman, is similar to but different from human–human dialogue. It is similar in the sense that people pursue a shared understanding: They actively work to make themselves understood. It is different in the sense that the communication abilities of computers are limited, which requires humans to adapt.

FSMs are an effective way to describe how dialogue is structured; however, they are limited to memory-free dialogue.

An FSM is a model of discrete computation applicable to dialogues.

A mode refers to the variation in the interpretation of a user's input according to an internal state. In a modeless dialogue, all inputs are possible in all states and their interpretation is always the same.

Dialogue can be described using models of computation from computer science. Such models include finite state machines (FSMs), pushdown automata, and Petri nets.

Affordance, which we discussed in Chapter 3, refers to how well users can interpret what actions are possible with a widget. Visibility is a handy related concept in design that underlies direct manipulation interfaces.

Mapping and feedback are crucial concepts for understanding a computer's turn.

Gulf of evaluation: This gulf refers to knowing how a perceived change in the computer has moved it closer to the intended goal state.

Gulf of execution: This gulf is about knowing what to do to bring about a desired state change in the computer.

Norman offered two central concepts to help us understand these cognitive efforts: the gulf of execution and the gulf of evaluation. These two concepts describe inferential breakpoints for users seeking to express their intentions and interpret feedback from the system, respectively.

A significant early theory of dialogue interaction is the seven-stage model of Norman [600]. It considers interaction as goal-directed, turn-based dialogue.

both the computer and the user may have initiative. For example, a pop-up window can be presented to confirm a risky selection. When there is a misunderstanding about the context of the dialogue, errors may happen, and the partners must recover from them.

both the computer and the human participate in establishing a shared context. The computer does not simply receive a message; it also communicates the effects of that message. Therefore, the design of feedback, affordances, and cues is central to dialogue-based interaction.

in dialogue, acts of communication are conditional: The meaning of a turn depends on the communication context.

the concepts of dialogue are applicable across modalities.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context.

Dialogue can be understood as computation, goal-directed action, communication, or embodied action. Each perspective provides specific methods for the analysis and design of dialogue.

Dialogue interaction includes speech-based and graphical interactions.

The core elements of dialogue are communication turns, the communication context, and turn interpretation.

Dialogue is about the organization of communication as a series of turns between communication partners.

The key idea in the dialogue view of interaction is the organization of communication as a series of turns. Dialogue evolves through communication turns between two or more partners. In one turn, an appropriate communication act is made by one partner based on the communication context. The act aims to get the other partner to do or understand something. This understanding then forms the context within which the other partner takes their turn.

Interaction may be viewed as a dialogue, that is, a conversation that occurs between two partners in a context for some purpose.

TAM posits that the intention to adopt a particular technology is driven by two kinds of perceptions: (1) how easy it is to use a system and (2) how useful it will be to use it [180]. Furthermore, the perceived ease of use affects the perceived usefulness: If technology is hard to use, it is less useful.

it is perfectly possible to have a program which is structured, modular, readable, flexible, self-documenting, maintainable, which performs its specified function, and which is a source of constant frustration and irritation to its users.

The theory of task–technology fit (TTF) can illuminate what users consider useful and how this affects their decision to adopt a particular technology. TTF refers to the ability of technology to support a task [197]. The capabilities of the technology should match the demands of the task and the skills of the individual; in this case, the fit is perfect.

Users actively repurpose tools to make them more personally usable and relevant. Design should support such repurposing. For example, Renom et al. [696] conducted a study on text editing using a novel user interface. They found that exploration and technical reasoning facilitate creative tool use.

One prominent definition of accessibility is given by ISO 9241-171, which defines it as 'the usability of a product, service, environment or facility by people with the widest range of capabilities.'

The second dimension, social acceptability, concerns whether interactions map well to the social norms and roles in the settings where they occur.

Acceptability has two main dimensions [591]. The first dimension, practical acceptability, includes costs, the reliability of the interactive system, and its compatibility with other systems. The perceptions of utility and usability may also influence the judgment of practical acceptability.

usability is multidimensional. This means that in most settings, a valid characterization of usability will need to employ several dimensions and measures.

usability is measurable, that is, it is possible to quantify usability based on users' behaviors or opinions.

usability is relational; it arises as an interplay between people, tasks (problems), and interactive systems (tools)

ISO 9241-11 definition... defines usability as the 'extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.'

One shorthand way of expressing this is that utility is 'whether the functionality of a system in principle can do what is needed' [591, p. 25]. In practice, whether people can do anything concerns—among other things—usability.

The utility of an interactive system concerns its match with the tasks of users. If the match is good, the tool has high utility; if the tasks that users want to do are not supported by the tool, the tool has low utility.

Usability concerns how easily computer-based tools may be operated by users trying to accomplish a task. Usability differs from utility. Usability concerns whether users can use the product in a way that makes it possible to realize its utility; utility is about whether the goal is important to the user. Ideally, the user can use the tool without unnecessary effort so that the use is direct, transparent, and unnoticeable.

Usability is one of the best predictors of users' willingness to adopt software. For example, the User Burden Scale is a questionnaire for measuring the felt burden in software use [806]. It consists of six subscales: difficulty of use, physical burden, time and social burden, mental and emotional burden, privacy burden, and financial burden.

However, self-attention alone is permutation-invariant, i.e., if we reorder the rows of X, then the mechanism has no built-in sense of which token came first. Since word order matters, we must inject positional information. We often add a position vector pt to the token embedding: h(0)_t = e(xt) + pt One classical choice for the positional encoding is called the sinusoidal positional encoding. pt[2k] = sin(t / 10000^{2k/d}), pt[2k+1] = cos(t / 10000^{2k/d}) The sinusoidal features give each position a distinct geometric signature across many frequencies. Nearby positions have related encodings while distant positions remain distinguishable. This lets the network reason about relative offsets.

For example, in control-theoretical analysis, we see the interaction as dynamically changing states.

For example, in interaction-as-rationality, courses of action emerge as a joint function of the user's goals and capabilities and the properties of the environment.

For example, statistical models, such as Fitts' law, describe a relationship that is considered a statistical determination.

For example, interaction-as-tool-use focuses on this idea (Chapter 19). Tools change people and their activities; in turn, this changes the tools.

An interactive AI system can possess an objective function it pursues when acting, for example, when correcting a character it flags as a typo.

In the case of particle physics, the propositions that make up a theory may concern the nature and behavior of particles.

HCI theories contain statements that link humans and technology and possibly some outcomes (e.g., poor usability, high user experience).

Text entry can also be seen as a task where different subtasks are shared between the human and the computer (Chapter 20).

One example is autocorrect, which automatically corrects typing errors while the user is typing. Another example is the use of word predictions, which allow the user to select a word from a set of word suggestions instead of typing out the word in full.

For example, text entry methods such as eye typing are designed to allow nonspeaking users with motor disabilities to enter text using their eye movements only.

Text entry is also a good example of tool use (Chapter 19). A text entry method is a tool that allows the user to communicate with someone or something, typically other people or a service, using asynchronous text messages and longer documents.

A fundamental goal of interaction is to provide users with tools that allow them to achieve goals that they would otherwise not have been able to achieve.

One goal of interaction is communicating intentions, for example, issuing commands to a computer, selecting graphical elements in image processing software, or entering text.

Accessibility is an extension of usability to ensure that as many people as possible find a tool easy to operate, regardless of individual capabilities.

Usability refers to the ease with which a tool is operated.

Utility captures how well a tool supports users in achieving their goals.

A model is a formally expressed set of propositions that follows some axiomatic system, such as algebra, logic, or a programming language.

The explanatory power of a theory refers to the empirical accuracy and coverage of the explanations offered by the theory.

A latent factor is something that affects observations about interaction without being directly observable.

Finally, in quantitative determination, interaction is described as a continuous unfolding of states.

In structural determination, the end results of the interaction are jointly determined by multiple causes that make up the whole.

In statistical determination, there is a stochastic relationship between the two entities.

In mechanical determination, an antecedent determines a consequent.

Propositions characterize entities and link them to other entities, some of which are conceptual.

In teleological determination, goals or purposes determine interaction in some way.

What happens in interaction is mutually determined by the human and the computer. In other words, what happens in interaction cannot be attributed solely to the human or the computer—the two must be considered together.

"Average movement time can be predicted as linear regression to the index of difficulty."

"The difficulty of selecting a target is proportional to its distance and inversely proportional to its width (index of difficulty)."

"If the user tries to increase speed, accuracy will be compromised, and vice versa: An increase in accuracy reduces speed."

For example, they can talk about information, difficulty, working memory, and so on.

A proposition is a claim about the world.

Phenomena in interaction are emergent, that is, they are not attributable to the user or to the computer alone.

Interaction also occurs in different contexts, including work, leisure, and in-between contexts such as commuting.

Interaction is a dynamic phenomenon that unfolds over time as users and computers influence each other.

What happens in interaction cannot be attributed solely to the human or the computer—the two must be considered together.

It has been used to describe individuals, groups, and communities using computers.

Such points about the origins of data and the processes of their collection are a key factor in civic text visualization. Indeed, a shift to emphasizing paradata can help draw attention to the representativeness of data.

On the other side of this spectrum, at the detail level, articulating nuanced information present in raw text data can enable civic leaders to peruse and sublimate critical insights.

In contrast, we could consider designing explicitly for multiple users. Doing so requires more than designing for different levels of expertise (see the following subsection for more on expertise) or designing for collaborative use, though both those things may be valuable in their own right. Rather, this dimension encourages accounting for the different types of relationalities that users may have with a system [cf. BB17].

Civic text visualizations similarly designed to foreground interpretation could help make clearer who is making these interpretive decisions, thereby highlighting the lack of neutrality and objectivity in data [DK20].