Researchers have done rich historical investigations of individual notations (e.g., [3, 70, 74, 76, 115]), but the more general mechanisms and patterns through which new notations are created and formalized are less understood.

We began with a diverse set of notations across five disciplines—music, dance, chemistry, physics, and computer programming—that had prior historical literature to draw upon in our initial analysis.

Since histories of specific notations tends to miss detailed, direct observations around the initial creation process, we complement this "macro" analysis with occasional references to experiment-based literature from experimental semiotics, communication theory, and cognitive science into how people use notations to ground communication, largely in lab studies.

we conducted a comparative historical analysis of the development of different notations which individually have been documented in prior literature. Specifically, we conduct a parallel comparative history which "seek[s] above all to demonstrate that a theory similarly holds good from case to case... [and where] differences among the cases are primarily contextual particularities against which to highlight the generality of the [theorized] processes"

Studying software teams, Cherubini et al. [34] found a "tendency to adopt informal, ad-hoc notations" and a "limited adherence to standards of any sort."

Studies are also conducted on various existing notating practices, usually in specific domains (e.g., how programmers draw diagrams to communicate ideas [34, 62, 63]).

Studies are also conducted on various existing notating prac-tices, usually in specific domains (e.g., how programmers drawdiagrams to communicate ideas [34 , 62 , 63 ]).

What seem today as obvious notations often have relatively short histories: for instance, arrows in diagrams emerged around the 18th century.

Notations are deployed and embedded throughout the process of HCI and software development.

Almost everything we do with computers involves notations.

Recent advancements in AI offer a new paradigm of subjectively inferring human intent from ambiguous input [165].

Recent advancements in AI offer a new paradigm of subjectively inferring human intent from ambiguous input.

These informal interactions can then lead to formal representations, but depend upon pre-existing formalisms known to both humans and AI.

Given that user interfaces are a form of notation, it is not surprising that their evolution closely follows the patterns we identified.

Seemingly 'obvious' notations to academics are also not obvious to everyone: e.g., about one-third of the U.S. and German populations have low literacy in reading data visualizations [51].

As current AI technologies rely upon, reproduce, and amplify established, dominant, already-formalized abstractions and notations in order to function

Many notations are culturally learned and inherited.

In this work, we lay some groundwork towards addressing thisquestion by presenting a parallel comparative historical analy-sis [ 129] of notation development across scientific, computing,and artistic disciplines.

From our analysis, we derive a set of initial implications for the design of future systems that create new abstractions (Section 5), including that notations primarily originate through linking metaphors and most often in a social—rather than a technical—context, and that notation design decisions around what to include as "meaningful" (and thus what to exclude) are often left implicit by inventors, but could be made explicit and become manipulable objects through reification [10].

Our work contributes to a longstanding dream of dynamic abstractions in HCI, where users can dynamically communicate and express themselves through notations (interfaces) that they are most comfortable with at the moment of expression, beyond ones predefined by developers [96, 143, 144, 148, 149].

Here we present suggestions for system designers, with concrete examples inspired by our patterns. These are just some interesting ideas that came to mind, rather than an exhaustive list.

Alongside the social stages of notation development above are three functional stages that emerge from reflection upon our analysis—descriptive, generative, and evaluative stages (borrowing terminology from Generative Theories of Interaction [11])

Our historical analysis suggests that, cognitively and socially, a notation proceeds by: (1) Enumerating dimensions of meaningful variation in the target domain, which proliferate as more situations are encountered or considered (whether by inventors or users) (2) Mapping dimensions of meaningful variation to perceptual channels of representation (3) Designing the notation to leverage perceptual affordances by visual analogy to embodied transformations like pouring cups or rotating shapes, and ensuring these "natural" manipulations hold meaning in the target domain

These stages form a spectrum and are not rigid boundaries. We clustered patterns into the most relevant stage for ease of presentation; however, patterns can be applicable across stages.

Our review identified many empirical patterns in the notation development process. We state each pattern, briefly describe it, and provide examples.

Our work heeds calls from HCI scholars for more historicism in the field, to better contextualize technology "within dynamic temporal processes of emergence, change, continuity, decline, disappearance, or revival," and to see "the past... as a repository of design knowledge and experience"

From our analysis, we derive a set of initial implications for the design of future systems that create new abstractions (Section 5), including that notations primarily originate through linking metaphors and most often in a social—rather than a technical—context, and that notation design decisions around what to include as "meaningful" (and thus what to exclude) are often left implicit by inventors, but

Our analysis identifies 33 patterns of how notations are created, evolved, and formalized over time, which are largely shared across histories and loosely categorized into three social stages of development (invention/incubation, dispersion/divergence, and institutionalization/sanctification) and three functional stages (descriptive, generative, and evaluative).

What about novel formalisms and notations? How are new abstractions created, evolved, and incrementally formalized over time—and how might new systems, in turn, be explicitly designed to support these processes?

How might we co-create a new notation with a machine, and thereafter communicate through that notation, even share out the notation to broader communities?

While current AI systems support "horizontal" translations from informal ideas to established notations, how should we ensure that the "vertical" process of creation—new notations, new abstractions—is also supported?

Why and when is formalizationhelpful, and when is it not?

What stages does a notation go through until itis considered sufficiently “formalized”?

How is a notation initially created? Arethere common steps or techniques?

Why does a notation emerge? What com-pels or necessitates the creation of a new notation?

How do humans ultimately develop new notations, new formalisms, and new abstractions, that they use to communicate with machines and each other?

The use of notation happens everyday in small ways, e.g., whenever people work together over a whiteboard or paper towards a joint objective. People jot down X's, boxes and arrows to stand-for concepts they are working through.

Recent AI systems promise a new set of informal interactions withcomputers through natural language and other notational forms.

Human-computer interactions have historically been mediated by formally-defined structures—such as command-line interfaces, graphical user interfaces, and programming languages—that provide an unambiguous mapping to an underlying formal model.

Yet humans collaborating together do not always defer to a pre-existing formalism. Instead, they can develop ad-hoc, new notations to ground their communication [22, 34, 143], treating notations more like malleable resources than rigid systems.

Traditional human-computer interaction takes place through formally-specified systems like structured UIs and programming languages.

SDT broadly differentiates three types of motivation [157]: Intrinsic motivation denotes activity pursued for its inherently interesting or enjoyable qualities. Extrinsic motivation refers to activity pursued for a separable outcome. Amotivation denotes the absence of intentional motivation, where a person may no longer be aware why they pursue an activity.

Basic psychological needs theory (BPNT) posits three basic psychological needs that energise organismic processes: competence, the feeling of having an effect; autonomy, a sense that actions are self-endorsed and performed willingly; and relatedness, a sense of reciprocal care, value, and belonging in relation to other social figures and collectives [158].

SDT is broadly organised into six mini-theories, whose underlying concepts are continuously developed, critiqued, and revised (e.g., [186, 190, 191]).

At its core, SDT is a scientific theory [163], in that it contains a number of empirically-testable propositions [199] that generalise across varied contexts, which serve to explain and predict the impact of certain events on motivation and wellbeing.

SDT is a psychological macro-theory of human motivation, growth, and wellbeing [47, 48, 163] that characterises humans as fundamentally active organisms.

Self-Determination Theory (SDT), a major psychological theory of human motivation, has become increasingly popular in Human-Computer Interaction (HCI) research on games and play.

Self-Determination Theory (SDT), a major psychological the-ory of human motivation, has become increasingly popular inHuman-Computer Interaction (HCI) research on games andplay.

To our knowledge, the first SDT research involving videogames [18] was conducted shortly after Deci's original formulation of CET [129] and investigated whether extrinsic rewards would reduce intrinsic motivation even for 'highly intrinsically motivating' activities such as videogame play. Videogames' intrinsically motivating qualities were also examined in early research on learning [e.g., 351]; however, focused examination of other core SDT concepts such as need satisfaction largely began much later [365].

Research on games and play in HCI (henceforth HCI games research), however, has continued to employ broad psychological theories as foundational work [417, 556]. One prominent example can be seen in self-determination theory (SDT) [481, 483], an influential theory of human motivation, which has provided HCI games research with propositions and concepts that can help explain motivational and experiential qualities of games and game-adjacent systems (e.g., gamification).

Psychological concepts and models have long been employed in human–computer interaction (HCI) to theorise the human user [88]. However, early applications of cognitive psychological theory did not develop into a coherent foundation of knowledge about human factors [89, 109, 455]—circumstances that Rogers [456, p. 22] attribute to "the stark differences between a controlled lab setting and the messy real world setting" for which interactive artefacts and systems are designed. The deployment of broad theory in HCI has subsequently declined in the intervening years [455, 456], and this sporadic progress in theory development in domains such as usability and user experience (UX) has been identified as a cause for concern [249, 314].

Psychological concepts and models have long been employed in human–computer interaction(HCI) to theorise the human user [88].

In HCI, frameworks function as a type of theoretical contribu-tion, often supporting ideation, design, and evaluation.

A father tells his daughter, “It’s time to put the tablet down.” Not wanting to stop using the tablet, butworried about the consequences of disobedience, the child finds herself in a dilemma. With a strokeof insight, she puts the tablet down on the table in front of her, and keeps playing with it. She can stilluse the tablet, and her father’s instructions were met. Technically.

An inital version is implemented in an ObservableHQ notebookwhere sets of lede sentences can be input.

First,it utilizes a neuro-symbolic pipeline to support Variation Theory-based counterfactual data generation.

Users teach models theirconcept definition through data labeling, while refining their ownunderstandings throughout the process

Alignment is a bilateral process; it refers not only to AI acting according to human intentions but also to humans better leveraging AI by understanding the mechanisms behind it [54].

Data labeling as a cognitive task—including defining a concept or determining how two similar objects may have different labels—requires both comparison and integration [62].

However, relying exclusively on existing examples is not ideal for tasks requiring nuanced understanding of user intentions, as these examples often fail to represent diverse and edge-case scenarios [31].

When training samples are scarce, model performance heavily depends on the quality of available training examples [15].

An important challenge in interactive machine learning, particularly in subjective or ambiguous domains, is fostering bi-directional alignment between humans and models.

In supervised and semi-supervised machine learning (ML) pipelines, labeled data is a vital component of training and validating models [46].

In the context of co-adaptive learning, supporting the intertwined evolution of both the user's understanding and the model's learning is crucial [16].

Machine teaching, a part of the human-in-the-loop approach, has been used as a process in which a human expert (the "teacher") provides guidance to a machine learning model to help it learn important and robust features for decision making [57].

A targeted approach in IML is machine teaching (MT) [60], an interactive framework that allows users to devise and select useful data for labeling, with the goal of teaching the model relevant features during training [7, 18].

Interactive ML (IML) methods, like active learning [3], continuously apply human feedback during model training to iteratively build and refine the model [35, 42, 43].

Reflection is recognised as vital for rigorous and responsible (HCI)research, yet it is often treated as secondary, hidden in the marginsof papers.

The goal of this meet-up is to create a space for CHI attendees to discuss and practise reflection in HCI research and design.

This meet-up invites CHI attendees to come together inthe META HCI community to explore how we as researchers andpractitioners reflect on our own practices – and how we might doso more intentionally.

Reflection is not limited to one subfield or methodology: it is a concern that cuts across the entire discipline.

We understand reflection as a multifaceted concept [6, 24, 25] with implications and relevance at different levels for researchers [2, 3, 8, 28]:

Reflection has been a recurring theme in HCI – from Schön's reflective practitioner [24] to Sengers et al.'s reflective design [25].

Importantly, reflection happens on multiple levels: as individuals questioning assumptions and choices, as groups working together in projects or labs, and as a community negotiating shared values, norms, and directions.

Reflection, as we envision it here, is not limited to standardised methods and codified practices, but also includes the hidden, vulnerable conversations about academic life.

Structures — reflection on the structures that condition HCI and our own standings within them: societal constructs (positions, values, power) shaping what problems are visible and whose knowledge is legitimised.

Practice — reflection on practice (processes): the ways we design, study, and collaborate [32].

Self — reflection on the self: a deliberate, structured evaluation of one's thoughts, feelings, and actions [7, 13].

We understand reflection as a multifaceted concept [6, 24, 25] with implications and relevance at different levels for researchers [2, 3, 8, 28].

While reflection is acknowledged as crucial for rigorous and responsible research, it often remains tacit and under-discussed.

Reflection has been a recurring theme in HCI – from Schön's reflective practitioner [24] to Sengers et al.'s reflective design [25]. However, it is seldom centred in our collective conversations [2].

As the community around augmented reading broadens and as possibilities continue to unfold, it is the purpose of this workshop to set up our community to drive innovation in a productive, desirable, and responsible way.

The landscape of technology for consuming information is changing rapidly. One mode of information consumption, reading, stands to see profound changes due to its ubiquity and frequency as a cognitive task.

Recent changes in the technological landscape are significantly changing the reading experience. AI has introduced many new possibilities for interfaces to augment or transform text to be more rapidly scanned, navigated, understood, and compared to other texts.

Reading is one of the most ubiquitous modes for consuming information.

Reading is one of the most ubiquitous modes for consuming in-formation.

Visualization is central to scientific discovery, yet authoring tools remain split between information and scientific visualization,and expertise in one rarely transfers to the other.

Butanalogy can also operate in mutual alignment1 analogies to reveal commonalities thatwere previously not obvious in either analog.

Projecting information from a well-understood domain can lend structure to an unfamiliar domain, as in:The mitochondria are the power supply for a cell.

Analogy is often thought of chiefly as a way to transfer knowledge from one situationto another, and indeed, it often serves that function.

We illustrate ourpoints with examples from adults and children, including examples from language evolu-tion, and across both perceptual and conceptual domains.

We propose that—bothin the history of language and in children’s learning—analogical processes are a majorway in which new relational abstractions are acquired

But the ear-lier we go in development, the less able children are to comprehend verbal explanationsof abstract ideas. In contrast, there is evidence that analogical comparison and abstractionprocesses are present in 7–9-month-old infants, and even earlier (Anderson, Chang, Hes-pos, & Gentner, under review; Ferry, Hespos, & Gentner, 2015).

Relational categories have been the focus of much recent research (Asmuth &Gentner, 2017; Gentner, 2005; Gentner & Kurtz, 2005; Goldwater & Markman, 2011;Markman & Stilwell, 2001; Ross & Murphy, 1999), in part because of their importantrole in conceptual learning and education (Goldwater & Schalk, 2016).

For example, carnivore andherbivore are abstract relational categories, while canine and feline are abstract entity cat-egories.

Relational cate-gories are categories for which the basis for membership is participation in a commonrelational structure; thus, they differ from the more studied entity categories, such as tulipand spoon, whose members share many intrinsic properties.

Our main focus is on relational abstractions, includingprinciples, rules, and schemas, as well as abstract relational categories.

For example, causal system is more abstract than posi-tive feedback system, which in turn is more abstract than the specific positive feedbacksystem by which the melting of polar ice causes lower reflectance of the sun’s heat, lead-ing in turn to more rapid melting.

Wetake the process of abstraction to be one of decreasing the specificity (and therebyincreasing the scope) of a concept.

Many such abstractions are expressed as rela-tional categories—categories like evidence, counterfactual, and proportion, and on a moremundane level, bargain, ally, and rescue.

Theassertions that make up abstract knowledge are variously referred to as schemas, rules,abstractions, principles, or overhypotheses

Abstract structured knowledge is a key feature of higher order cognition (Gentner &Medina, 1998; Hummel, 2011; Markman, 1999; Tenenbaum & Griffiths, 2001).

it is not enough to consider the distribution of examples given to learn-ers; one must consider the processes learners are applying

Wepropose that analogical generalization drives much of this early learning and allows children togenerate new abstractions from experience

contrary to the general assumption,maximizing variability is not always the best route for maximizing generalization and transfer

Gebreegziabher et al. [24] argued that counterfactual generation that follows the principles of VT allowed the introduction of discriminatory variance for the model to learn on.

Building on methods proposed in PaTAT [24], Mocha first generates human-readable neuro-symbolic pattern rules from partially labeled text data for classification.

These theories have proven insightful for understanding how humans grasp and compare concepts, shaping the development of human-AI collaboration systems for sensemaking [29], hypothesis testing [2], as well as model training [24].

Both systems enabled users to quickly identify variations and patterns within the data and support exploration and hypothesis testing.

The last two prior works also combine Variation Theory (VT) and SAT together, as we did (i.e., a corollary of SAT referred to as Analogical Transfer/Learning Theory).

In line with previous work, Mocha aims to support a user's efforts in the disambiguation of concepts through structural comparisons of counterfactual data in the context of machine teaching.

Engineering refers to the use of technical principles, such as mathematics, science, and technical know-how, to realize a design that best meets a given set of expectations, which are typically captured in a requirements specification.

Designing is the process of arriving at a plan, specification, prototype, system, or service—a design. In HCI, this often means designing a user interface and relevant parts of the underlying interactive system.

HCI focuses on people who use an interactive system or are affected by its use. This focus is often called being user-centered or human-centered to contrast it with a focus on the technology itself [423, 604].

Finally, interaction often involves co-adaptation between people and computers [646], meaning that both the user and the system learn and adapt to each other during interactions.

Interaction is, in other words, not a property of the system design or the user but something that emerges when they influence each other.

The development of technology for interactive computing systems has been an important driver behind the widespread adoption of computing we have witnessed in the last 50 years.

In HCI, evaluation refers to the application of some systematic methodology to attribute human-related values to an artifact, prototype, system, or process. Examples of such attributes include performance, experience, safety, and ethical aspects, such as the avoidance of bias or harm.

Programmability lends computers their power as tools. Computer programs can decompose complex activities into sequences of much simpler operations.

A special part of a computing system is the user interface. It is the part that the user can see and utilize to control the computer. Through the user interface, users can provide input and instructions to a computer and receive feedback from it. In short, the user interface enables interaction with a computer.

Beforethen, the terms man–computer interaction and man–machine interaction had been in use sincethe early 1960s [e.g., 476, 588].

In multitasking, tasks compete for limited sensory, motor, and central (cognitive) capacities

Visual objects that are unique in their visual primitives attract user's attention.

HCI phenomena span eye movements, emotional reactions, aesthetic experiences, social interactions, and organizational structures; they also span behaviors from the millisecond level to changes in the use of interactive systems over decades as well as the individual, group, and societal levels.

Interaction is a concept that is fundamental in HCI and specific to this field [357]. Intuitively, it refers to the reciprocal influence between people and an interactive system that takes place through the user interface.

Users continuously adapt their social behavior to compensate for the lack of social cues in computer-mediated communication

Users' performance in providing input to a computer is limited by a speed–accuracy trade-off

A mental model captures how people understand something. For instance, people have vastly different beliefs about how calculators work [598]. These beliefs can explain the errors and the issues they face when using calculators.

At the core of this revolution was the ability to flexibly define and execute computer programs—sequences of logical operations executed by a computer.

Interactive systems are tools that help users achieve their goals.

The remarkable efficiency, flexibility, and scalability of computers as tools boil down to the concept of a programmable machine capable of interpreting computer programs.

Programmability lends computers their power as tools.

They enhance our physicalabilities and are central to many intellectual activities, such as writing, mathematics, and account-ing.

From fishing nets to drilling machines, tools are vital to human ability.

A key technical construct in HCI is the user interface. It refers to the parts of an interactive system that the user comes into contact with or that in other ways shape the user's perception of the system.

In such areas, it is commonplaceto make assumptions about users without always grounding them in empirical observations ortheory.

Human–computer interaction is a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them.

For example, an expert in HCI4D (HCI for development) described the challengesfaced in non-Western contexts as follows [184, p. 2228]:We need to address the everyday problems of people. Most people don’t know how to scroll, navigate.We need to do basic HCI work to make text larger. Also, time of day is the most prominent thing on [aphone’s] screen. Let’s replace that with the amount of airtime you have left. We need to improve uponwhat we built yesterday rather than doing novel interventions or focusing on the future.

Second, the computer is among the most complex tools humans have devised.

How should all this computing powerbe used and for what?

How can people with different goals and capabilities, and in different contexts, be able to use computingproductively, enjoyably, and safely?

In HCI, evaluation refers to the application of some systematic methodology to attribute human-related values to an artifact, prototype, system, or process.

The development of technology for interactive computing systems has been an important driverbehind the widespread adoption of computing we have witnessed in the last 50 years.

A special part of a computing system is the user interface. It is the part that the user can see and utilize to control the computer.

Programmability lends computers their power as tools.

It is an egocentric fallacy to assume that others are like us—to attempt to explain other people by reference to one's own experience.

A study of large-scale web-clicking data employed this theory to explain why certain distributions of web page hits emerge on web sites. Huberman et al. [362] proposed a mathematical model that assumes that at any page, users decide to continue clicking as long as its information scent exceeds some threshold. This information scent can be computed using information foraging theory (IFT).

IFT proposes that information-seeking behavior develops to maximize the rate of information gained per unit of time or effort invested. Note that the term information does not refer to the information-theoretic concept but to subjective interest; here, information means anything that users find interesting.

The roots of IFT lie in optimal foraging theory, originally proposed in biology, which describes the hunting and food search behaviors of animals.

A solution strategy—or

Computational rationality is a theory and a modeling approach rooted in bounded rationality and bounded optimality. Recent applications include typing (Figure 21.7), pointing, driving, multitasking, menu selection, and visual search.

Theories of rationality have increased our understanding of how users fail to be optimal.

MDP is a formalism that originates from studies of sequential decision-making in artificial intelligence and operations research. Instead of the choice between n actions, MDP deals with environments where rewards are delayed (or distal). This requires an ability to plan actions as part of sequences instead of one-shot choices.

Information scent refers to a user's intuition that a cue in the interface represents the information needed. It is an estimation of relevance based on a proximal cue.

IFT proposes that information-seeking behavior develops to maximize the rate of information gained per unit of time or effort invested.

Information foraging refers to information-seeking activities such as navigating, exploring, comparing, searching, or manipulating information contents in an information space.

A payoff refers to the benefits that are left after the costs have been subtracted.

The space of all states, actions, and rewards is called the environment.

Costs are negative rewards that the user incurs for transitioning between states or for being in states that are not good for them.

Visual statistical learning is a research topic in perception that studies how the statistical distribution of our environments affects the deployment of gaze.

bounded rationality states that we are only rational to the extent allowed by the involved constraints, or bounds.

An agent is an actor with the ability to choose actions in pursuit of some goal or reward.

To state that a user's choice is rational means that it is selected with the expectation that it yields the highest utility out of the available options.

It assumes that human long-term memory evolved to help survival by anticipating organismically important events. It is evolutionarily important to remember things that are important for survival. Therefore, the expected value of remembering a thing in the future should affect the probability of recalling it.

According to rational analysis, behavior is sensitive to the statistical distribution of rewards in the environment that a user has experienced. Users learn the way rewards are distributed through continued exposure to an environment and adapt their behavior accordingly. A user's behavior is rational because it is tuned to the distribution of rewards in the environment—the ecology.

The theory assumes that users are 'computationally rational': When picking an action—or deciding how to get from the present state to a state with positive rewards—users are as rational as their cognition allows. Users act based on their often inaccurate and partial beliefs, which they have formed via experience.

Computational rationality is a theory and a modeling approach rooted in bounded rationality and bounded optimality. Recent applications include typing (Figure 21.7), pointing, driving, multitasking, menu selection, and visual search. Its core assumption is that users act in accordance with what they believe is best for them.

Rational analysis is a theory of rational behavior proposed by Anderson and Schooler [21]. It examines the distribution of rewards in the environment to explain how users adapt their behavior. According to rational analysis, behavior is sensitive to the statistical distribution of rewards in the environment that a user has experienced.

They share a focus on the emergence of interactive behavior; in other words, they predict how users choose to behave in certain given circumstances.

Utility refers to the agent's consideration of positive and negative rewards when deciding how to act.

Theories of rationality can be used to inform the design of information environments, addressing considerations such as how to distribute and shape information.

These four theories differ in the factors they include and how the agent's decision-making problem is formulated. As such, the theories differ in how easily they help us find a solution to the user's decision-making problem.

Theories of rationality can make quantitative predictions on user behavior in settings where the user's environment and goals (rewards) are well known.

Theories of rationality do not describe what a user has done but ask what that user could have done. Rational behavior refers to behavior that seeks to maximize the expected utility to the user.

Descriptive theories attempt to capture causes behind behaviour that, from a normative perspective, may appear irrational. This view enables predictions of user behaviour in real-world circumstances.

Descriptive theories attempt to capture causes behind behavior that, from a normative perspective,may appear irrational. This view enables predictions of user behavior in real-world circumstances.

bounded rationality states that we are only rational to the extent allowed by the involved constraints, or bounds.

The term satisficing is used to describe how users tend to behave when facing a complex decision-making problem. It refers to settling on a satisfactory but not optimal solution in the normative sense.

The concept of rationality has its roots in economics, where it was developed to study how peo-ple should act in economic decision-making. In such settings, the idea is that people reach theirgoal, such as maximizing their return, by maximizing utility.

This chapter introduces the notion of rationality. This allows us to provide explanations forinteractive behavior due to users attempting to make the most out of the choices available to them.

The author wants to augment the formula to explain the meaning of the terms on either side of the arrow—first

they often benefit from being augmented with descriptive elements, such as labels describing the meaning of an expression or colors linking an identifier to its description in the text.

In this walkthrough, the author is trying to add labels to the formula V(s_t) ← R_t to describe the meaning of its terms in an article they are writing.

Our design was motivated by two major goals for notation authoring. These goals followed from recent studies of notation augmentation [30, 71] and conversations with scientists who had experience writing notation in instructional materials and research communications (4 professors, 2 graduate students, R1–6).

authors sometimes fall back on graphical editors like Google Slides, PowerPoint, Adobe Illustrator, and Mathcha to augment formulas.

Authors of typeset formulas augment those formulas to make them easier to understand.

We define the key projections as markup (in this case, LaTeX), an annotatable render, and a structure hierarchy view. Augmentations are made easy to invoke, and projections are kept synchronized and co-present so that authors can shift between representations as is expedient to them.

the challenge of using these tools is that annotations are unmoored from the structure of the formula and must be redone whenever the formula changes. Authors must perform precision positioning and sizing operations that could be inferred from the coordinates of the augmented expressions.

these markup languages can require cumbersome and error-prone editing, arising from the intermixing of annotation markup with the underlying formula. Participants in a study by Wu et al. [71] identified difficulty with debugging nested braces and locating markup to edit.

lab study participants frequently made errors related to incorrectly matched braces when using a LaTeX baseline to augment formulas.

Authors in Head et al. [30] described that "code gets horrible looking" as macros are added to it to specify augmentations.

FreeForm, a projectional editor wherein authors can augment formulas—with color, labels, spacing, and more—across multiple synchronized representations. Augmentations are created graphically using direct selections and compact menus. Those augmentations propagate to LaTeX markup, which can itself be edited and easily exported.

FreeForm is a projectional editor optimized for notation augmentation. This paper defines the key projections for the text: textual LaTeX, a formula render with tree-aware selections, and a property/hierarchy view.

The key ideas are that authors can interact with different projections of the formula at will, and graphical edits are ultimately connected back to the formula via markup.

Authors of typeset formulas augment those formulas to make them easier to understand.

Ply offers this LLM-supported program decomposition supported by visualization and parameterization UIs, permitting users to use interactions beyond chat to compose their programs incrementally.

designing complex behavior can be a difficult programming task, and program representations in end-user programming tools may not be well-suited for heavy programs.

Ply allows users to develop, test, and tweak program components, exploring possibilities for how data can be transformed and composed to discover and achieve goals. This style of programming can support many use cases, even those not traditionally considered in the trigger-action programming model.

Through the combination of these features, Ply allows users to develop, test, and tweak program components, exploring possibilities for how data can be transformed and composed to discover and achieve goals.

Frequently, code-generation systems focus on building and then refining a full working application, using visibility of the full underlying code as a fallback when users need to build understanding of the generated program.