JSONSchemaBench: Structured Output Benchmark for Language Models

Core Contribution

• Benchmark introduction: JSONSchemaBench comprising 10K real-world JSON schemas for evaluating constrained decoding frameworks

  "We introduce JSONSchemaBench, a benchmark for constrained decoding comprising 10K real-world JSON schemas that encompass a wide range of constraints with varying complexity"

• Three-dimensional evaluation framework assessing efficiency, coverage, and quality of constrained decoding approaches

  "We present an evaluation framework to assess constrained decoding approaches across three critical dimensions: efficiency in generating constraint-compliant outputs, coverage of diverse constraint types, and quality of the generated outputs"

• Six frameworks evaluated: Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini

  "We evaluate six state-of-the-art constrained decoding frameworks, including Guidance, Outlines, Llamacpp, XGrammar, OpenAI, and Gemini"

Key Findings

Efficiency Results

• 50% speedup potential from constrained decoding over unconstrained generation

  "Constrained decoding can speed up the generation process by 50% compared to unconstrained decoding"

• Guidance achieves best throughput through guidance acceleration technique

  "Guidance achieves even higher efficiency, which it accomplishes by fast-forwarding certain generation steps with its guidance acceleration"

• Compilation time varies significantly: Outlines has "significantly higher compilation time" compared to Guidance and Llamacpp which have "minimal grammar compilation time"

Coverage Results

• Coverage disparity: Best framework supports twice as many schemas as worst

  "Frameworks demonstrate significant differences in their actual support for real-world JSON schemas, with the best framework supporting twice as many schemas as the worst"

• Guidance leads empirical coverage on 6 out of 8 datasets

  "Guidance shows the highest empirical coverage on six out of the eight datasets"

• Compliance rate critical: Guidance demonstrates "highest compliance rate across all datasets, making it the most reliable option for ensuring schema compliance"

Quality Results

• Performance improvement: Constrained decoding improves downstream task accuracy up to 4%

  "Constrained decoding consistently improves the performance of downstream tasks up to 4%, even for tasks with minimal structure like GSM8k"

• Guidance best for quality: "Guidance consistently delivers the best performance across all tasks, with approximately a 3% improvement over the LM-only approach in every task"

Core Concepts

Coverage Definitions

• Declared Coverage: Schema processed without rejection or runtime errors

  "A schema is considered declared covered if the framework processes the schema without explicitly rejecting it or encountering runtime errors such as exceptions or crashes"

• Empirical Coverage: Experiments show framework produces schema-compliant outputs

  "A schema is considered empirically covered if our experiments show that the constraints generated by the framework result in LM outputs that are schema-compliant"

• True Coverage: Constraints precisely equivalent to JSON Schema definition

  "A schema is considered truly covered if the framework produces constraints that are precisely equivalent to the original JSON Schema definition"

• Compliance Rate: Ratio of empirical to declared coverage

  "Compliance Rate = C_Empirical/C_Declared... estimates the reliability of the constrained decoding framework in guaranteeing compliance given it accepts a given schema"

Failure Modes

• Over-constrained: Framework rejects valid JSON instances

  "A framework is over-constrained if it rejects JSON instances that are valid according to a given JSON Schema. This means the engine is too strict and excludes outputs that should be allowed"

• Under-constrained: Framework allows invalid JSON instances

  "A framework is under-constrained if it allows JSON instances that are invalid according to a given JSON Schema. This means the engine is overly permissive and allows outputs that should be rejected"

Evaluation Methodology

Efficiency Metrics

• Grammar Compilation Time (GCT): Time spent on grammar compilation
• Time to First Token (TTFT): Time from generation start to first token
• Time per Output Token (TPOT): Average time per token after first

Dataset Composition

• 10 dataset suites with varying complexity:

  "We split the data into five collections based on the schema size: trivial, small, medium, large, ultra"

• Sources include:

• GitHub schemas (6,000 from Baazizi et al., 2021)
• JSON Schema Test Suite (official test cases)
• Schema Store (largest independent collection)
• GlaiveAI function calling dataset (2,000 schemas)
• Kubernetes configuration files
• Washington Post ANS specification

Experimental Setup

• Model used: Llama-3.1-8B-Instruct for efficiency, Llama-3.2-1B-Instruct for coverage
• Hardware: Single NVIDIA A100-SXM4-80GB GPU with AMD EPYC 7543 CPU
• Validation: jsonschema Python library with Draft2020-12, format checks enabled
• Generation: Greedy decoding, zero temperature, single run, 40s timeouts

JSON Schema Test Suite Analysis

• 45 categories covering JSON Schema features (43 after filtering)

  "The test suite organizes its test cases into 45 categories, each of which corresponds to a feature of JSON Schema"

• Guidance dominates:

  "Guidance outperforms other engines at all coverage levels, achieving full coverage on 13 categories and moderate coverage on 21"

• Failure patterns vary:

  "Outlines, Llamacpp, and Guidance follow a consistent failure pattern, with most errors occurring during compilation and over-constrained failures being more frequent than under-constrained ones. In contrast, XGrammar minimizes compilation errors but shows the highest number of under-constrained failures"

Quality Evaluation Tasks

Three Reasoning Tasks

• Last Letter: CoT reasoning + answer in a-z

  "Input: Ian Peter Bernard Stephen Output: nrdn"

• Shuffle Objects: CoT reasoning + answer in A-E

  "Input: Sequence of exchanges among individuals + choices Output: A-E"

• GSM8K: CoT reasoning + answer as integer/float

  "Input: Basic calculation problems Output: Number, e.g. 8"

• JSON structure: All tasks use {"reasoning": <reasoning>, "answer": <final answer>} format

Problem Context

Motivation

• Machine-oriented applications require structured outputs

  "Unlike traditional natural language processing (NLP) tasks where the output is aimed at review by humans, output in these applications is often consumed by machines such as controller and service APIs"

• Probabilistic nature problematic:

  "However, the LM generation process is probabilistic and does not provide guarantees on the output's structure, making it challenging to deploy LMs in applications requiring structured inputs and high reliability"

• JSON Schema as standard:

  "JSON Schema has emerged as a key specification language for constrained decoding... Commercial LM providers, such as OpenAI, have embraced constrained decoding by incorporating support for JSON Schema directly into their APIs"

Research Gap

• Evaluation under-explored:

  "The evaluation of constrained decoding remains an under-explored topic, with no consensus on what defines the effectiveness of constrained decoding"

• No framework comparisons: Prior studies "fail to provide comparisons across different constrained decoding frameworks"

• Limited benchmarks: Previous benchmarks "narrowly focused on specific tasks or rely on formal-grammar–based artificial setups, that have unclear relevance to real-world use cases"

Technical Implementation

Constrained Decoding Algorithm

• Core mechanism: Token masking at each generation step

  "Constrained decoding intervenes in the decoding process of LMs by masking out invalid tokens based on given constraints and prefix tokens"

• Algorithm steps: Update constraint state → compute mask → calculate logits → apply mask → sample token → append to output

Addressing Coverage Bias

• Intersection-based metrics: Calculate efficiency only on schemas all engines support

  "To ensure fairness, we calculate efficiency metrics on the intersection of covered instances across all engines"

• Rationale: "Engines with lower coverage often process simpler, shorter schemas, which naturally compile and generate faster"

Key Tools & Frameworks

Open Source Engines

• Guidance (Guidance AI, 2023): Dynamic constraint computation, token healing
• Outlines (Willard & Louf, 2023): Regex-based constraints
• Llamacpp (Gerganov & al., 2023): Grammar module, GGML BNF
• XGrammar (Dong et al., 2024): Concurrent compilation with pre-filling

Closed Source APIs

• OpenAI: JSON Schema support in API
• Gemini: Limited schema support (<1% on some datasets)

Validation & Testing

• jsonschema Python library (Berman, 2025): Draft2020-12 validation
• JSON Schema Test Suite (JSON Schema Org, 2024): Official correctness tests
• Bowtie (2025): Cross-implementation comparison tool

Related Work

JSON Schema Collections

• Baazizi et al., 2021: 6,000+ schemas from GitHub repositories
• Attouche et al., 2022: Witness generation algorithm evaluation
• Schema Store: Largest independent JSON schema collection

Constrained Decoding Research

• Early methods:
• Deutsch et al., 2019: General-purpose constrained sequential inference
• Shin et al., 2021: Constrained language models as semantic parsers

• Scholak et al., 2021: PICARD for incremental parsing

• Optimization frameworks:

• Beurer-Kellner et al., 2023: Prompting as programming query language
• Kuchnik et al., 2023: ReLM validation
• Zheng et al., 2024: SGLang structured execution

Quality Concerns

• Geng et al., 2023: Grammar-constrained decoding without finetuning
• Tam et al., 2024: Study on format restriction impacts
• Kurt, 2024: Response to performance decline concerns, argues issues stem from "inadequate prompting, insufficient contextual information, and poorly crafted schemas"
• Geng et al., 2024: Tokenization ambiguity issues

Applications

• Yao et al., 2023: Web navigation (ReAct)
• Polak & Morgan, 2024: Materials data extraction
• Schick et al., 2023: Toolformer for tool use

Advanced Features

Optimization Techniques

• Grammar caching: Reuse compiled constraints
• Parallel execution: "Mask computation can run in parallel with the LM's forward pass, and grammar compilation can be performed concurrently with pre-filling computations"
• Speculative decoding: Constraint-based approach (GuidanceAI, 2024)
• Token healing: Guidance's technique for handling tokenization boundaries (GuidanceAI, 2024)

JSON Schema Features

• Most common keywords: type, properties, required, description, items, enum
• Format constraints: date-time, email, uri, uuid (each complex to implement)
• Advanced features: $defs, $ref, if-then-else, minItems, maxItems, pattern matching

Limitations & Future Work

Current Challenges

• Under-constraining tradeoffs:

  "Under-constraining effectively delegates responsibility to the LM, which may produce valid output despite a lack of strict constraints... requires careful implementation and transparency to ensure reliability"

• Feature representation: "Not all features are equally represented in real-world schemas... strong or weak performance on specific features can have disproportionate impacts depending on their prevalence"

• Test suite limitations: "No straightforward correspondence between test suite performance and empirical coverage"

Aspirational Goals

• GitHub-Ultra dataset: Retained "as an aspirational target for future advancements" despite being too hard for current frameworks

• Format validation: Paper excluded 'format' tests but notes "We hope to extend this work to include these optional tests in a follow-up"

• Formal verification: True coverage requires "formal verification method that is capable of exhaustively comparing the schema's semantics against the framework's implementation"

Repository & Resources

• Benchmark release: https://github.com/guidance-ai/jsonschemabench
• JSON Schema specification: Draft 2020-12 (Wright et al., 2022)
• Model used: Llama-3.1-8B-Instruct (Grattafiori et al., 2024)

https://youtube.com/watch?v=TAQ7yBLRZ3U&feature=shared

Certainly! Here’s a detailed summary and key insights from the YouTube talk “Use.GPU - Declarative/Reactive 3D Graphics by Steven Wittens #LambdaConf2024” (link to video):

Overview

Steven Wittens introduces Use.GPU, a TypeScript library for driving WebGPU with a declarative and reactive programming model. The talk explores the motivation, design, and technical underpinnings of Use.GPU, emphasizing productivity, maintainability, and the bridging of web and graphics paradigms.

Key Topics Covered

1. The Problem with Traditional 3D Graphics Development

• High Complexity & Maintenance Cost: Building custom 3D graphics (e.g., configurators, data visualizations, CAD apps) is often slow, expensive, and results in code that’s hard for teams to maintain.
• Specialization Barrier: The field is so specialized that many companies avoid using advanced GPU graphics due to the expertise required.

2. The Permutation Problem

• Example: A 3D house configurator requires manually assembling assets and coding every possible combination of options, leading to exponential complexity.
• Customization Pain: Existing visualization libraries (like Deck.gl) are hard to deeply customize without forking and maintaining complex codebases.

3. The Web vs. Graphics Divide

• Graphics World: Driven by games/CAD, large teams, offline delivery, monolithic codebases, and focus on rendering performance.
• Web World: Driven by SaaS, small teams, continuous delivery, focus on compatibility, composition, and reuse.
• Different Priorities: These differences make it hard to bring GPU graphics into mainstream web development.

4. Live: A React-like Runtime

• What is Live? A React-inspired, incremental, and reactive runtime that allows for declarative UI and graphics code.
• Key Features:
• Incremental updates: Only re-executes code in response to changes.
• Implicit, one-way data flow.
• Declarative side effects: Auto-mounting and disposal.
• Enables features like undo/redo and multiplayer state management.
• Unique Twist: Live allows data to flow back from child to parent components—something not possible in React—which is crucial for certain graphics/data workflows.

5. Use.GPU: Declarative WebGPU

• Goal: Make GPU graphics as easy to use and maintain as modern web UIs.
• Approach: Use familiar JSX-like syntax and React-style components to describe 3D scenes and behaviors.
• Incremental Rendering: The system is designed as if rendering one frame, and only reruns necessary parts for interactivity/animation.
• Bridging the Gap: By combining Live’s reactive model with WebGPU, Use.GPU makes advanced graphics accessible to web developers.

6. Technical Insights

• Immediate vs. Retained Mode:
• Immediate mode (e.g., Canvas): Easy but doesn’t scale for complex interactivity.
• Retained mode (e.g., GPU): More efficient but much harder to program and maintain.
• GPU as a Pure Function Applicator: The challenge is efficiently feeding unique data to millions of parallel shader invocations, with memory bandwidth as a key constraint.
• Use.GPU’s Innovation: Abstracts away much of the boilerplate and complexity, letting developers focus on high-level structure and reactivity.

Why This Matters

• Productivity: Use.GPU aims to democratize GPU programming for web developers, reducing the need for deep graphics expertise.
• Maintainability: Declarative, reactive patterns make complex interactive graphics more maintainable and composable.
• New Possibilities: Opens the door for more sophisticated, interactive, and visually rich web applications.

Further Resources

• Use.GPU Project: GitHub link (if available, check video description) (replace with actual link if different)
• Steven Wittens’ Website: acko.net (famous for its interactive header)
• LambdaConf: LambdaConf 2025 Tickets

TL;DR

Use.GPU is a new TypeScript/WebGPU library that brings React-style declarative, reactive programming to 3D graphics in the browser. Built on the “Live” runtime, it enables maintainable, high-performance graphics apps with familiar web development patterns—potentially revolutionizing how interactive graphics are built on the web.

If you want a specific section of the talk summarized, or code examples from Use.GPU, let me know!

Citations: [1] watch?v=TAQ7yBLRZ3U https://www.youtube.com/watch?v=TAQ7yBLRZ3U

UDA (Unified Data Architecture) at Netflix - Summary

Problem Statement

• Netflix faces growing complexity as offerings expand across films, series, games, live events, and ads

• Core business concepts (actor, movie) are modeled independently across multiple systems with no coordination

  "Each system models these concepts differently and in isolation, with little coordination or shared understanding."

Key Challenges Addressed

• Duplicated and Inconsistent Models — Teams re-model same entities in different systems with conflicting definitions
• Inconsistent Terminology — Different terms for same concept, or same term for different concepts

• Data Quality Issues — Discrepancies and broken references hard to detect across microservices

  "While identifiers and foreign keys exist, they are inconsistently modeled and poorly documented"

• Limited Connectivity — Cross-system relationships effectively non-existent

What is UDA?

• Foundation for connected data in Content Engineering

  "UDA enables teams to model domains once and represent them consistently across systems — powering automation, discoverability, and semantic interoperability."

Core Capabilities

1. Register and connect domain models — Formal conceptualizations of federated business domains
2. Catalog and map domain models to data containers — GraphQL resolvers, Data Mesh sources, Iceberg tables
3. Transpile domain models into schema languages — GraphQL, Avro, SQL, RDF, Java while preserving semantics
4. Move data faithfully between containers — Automatic handling of data transformation between systems
5. Discover and explore domain concepts — Via search and graph traversal
6. Programmatically introspect the knowledge graph — Using Java, GraphQL, or SPARQL

Technical Foundation

• Knowledge Graph — Built on RDF and SHACL

  "We chose RDF and SHACL as the foundation for UDA's knowledge graph"

• Named-graph-first information model — Each named graph conforms to a governing model

Upper Metamodel

• Language for formally describing domains and their concepts

  "Upper is the metamodel for Connected Data in UDA — the model for all models"

• Key properties:

  • Self-referencing — Models itself as a domain model
  • Self-describing — Defines the concept of a domain model
  • Self-validating — Conforms to its own model
• Domain models expressed as conceptual RDF, organized into named graphs
• Enables projections to GraphQL, Avro, Iceberg, Java

Mappings

• Data connecting domain models to data containers

  "A Mapping connects nodes in a subgraph of the domain model to nodes in a subgraph of a container representation"

• Enable discovery by walking knowledge graph to find concept materializations

• Support intent-based automation for data movement

Projections

• Produce concrete data containers (GraphQL schemas, Data Mesh sources)

  "Each projection is a concrete realization of Upper's denotational semantics, ensuring semantic interoperability across all containers"

• Transpilation targets: GraphQL, Avro (Data Mesh flavor)

• Some containers auto-populated (Iceberg Tables) via Data Mesh platform

Early Adopters

Primary Data Management (PDM)

• Single place for business users to manage controlled vocabularies

• Uses SKOS (Simple Knowledge Organization System) W3C standard

  "PDM uses Domain Models to integrate SKOS into the rest of Content Engineering's ecosystem"

• Auto-generates: UI, Domain Graph Service, GraphQL APIs, Data Mesh pipelines, warehouse data products

Sphere

• Self-service operational reporting tool

  "Instead of specifying exact tables and join keys, users simply can search for familiar business concepts such as 'actors' or 'movies'"

• Uses UDA knowledge graph for query generation via graph traversal

• Identifies join strategies, boundaries, and islands in data landscape

Future Directions

• Protobuf/gRPC projections
• Materializing knowledge graph of instance data
• Solving Graph Search challenges

Key References

• Netflix Enterprise GraphQL Gateway
• Data Mesh Platform
• Data Movement via Data Mesh
• Domain Graph Service Framework
• UDA GitHub Repository

Can coding agents self-improve? - Summary

Core Thesis

• Inference-time vs training-time self-improvement:

  "If you could never update the weights, how would you have the model increase its performance on a specific task? I think of that as inference-time self-improvement"

• Key finding:

  "The good news is that GPT-5 is a very good model for building developer utilities. The bad news is that it hates using the tools it creates! As it told me 'I'll be honest - I didn't need any of them.'"

Experimental Setup

• Testing methodology: Asked models (GPT-5 and Opus 4) to:

  1. Build tools they thought would help productivity
  2. Attempt tasks using those tools
  3. Self-reflect on tool improvements

• Core hypothesis:

  "The most valuable use case of coding agents is being a vessel for LLMs to extract value out of their own latent spaces"

• Comparison: Also tested Gemini 2.5 Pro and GPT-4.1, but focused on GPT-5 vs Opus 4 as only models that could keep up

Tool Creation Results

Task Manager Tool

• GPT-5 implementation features:

  "Uses WAL to avoid issues with multiple agents writing at the same time"

  "Uses a graph of dependencies to prioritize tasks"

  "Created an append-only events stream that lets any agent see what every other agent is doing with good keywords like impact_conflict"

• Opus 4 limitations:

  "didn't pick up on the notifications / stream functionality to keep everyone in sync"

Codebase Standards Tool

• Purpose:

  "analyze a codebase and extract different heuristics on how code should be written. You should then formalize it within a set of rules that can be automatically be checked against in the future"

• Quality comparison:

  "I've found the GPT-5 one to be much more nuanced than Opus"

Self-Proposed Tools

GPT-5 tools (16 total):

• doctor, bootstrap, code-map, csearch, tasks-graph, impact, seed, repro scaffold, e2e, preflight, preflight-smol, broker, flake, codemod, triage, trace, runbook

• Design philosophy:

  "GPT-5 built all of them as unix utilities that are easy to use via cli"

• Characterization:

  "GPT-5 was building utilities it could use itself without being too opinionated"

Opus 4 tools (10 total):

• Context Analyzer, Cross-Platform Test Generator, Implementation Proposal Analyzer, Full-Stack Change Impact Analyzer, Bug Pattern Recognition Engine, Security & Permission Auditor, Multi-Platform Feature Implementer, API Integration Assistant, Performance Optimization Toolkit, Task Complexity Estimator

• Design approach:

  "all meant to be run as python some_tool.py"

• Characterization:

  "Opus 4 was building tools that accomplish tasks and have a bit of anthromorphized feeling"

Task Execution Results

Test Task

• Project: smol-podcaster migration from Flask to FastAPI + Next.js

• Task complexity:

  "the task I tried would take me 4-5 hours to do"

• Performance:

  "Both models were almost able to one-shot the task"

Tool Usage Discovery

• First attempt: Both models completed task successfully but

  "They both said they did not use ANY of the tools they had built, except for the tools they were already familiar with"

• GPT-5 second attempt response:

  "Short answer: no — I didn't use the devtools in this run. [...] The failures were runtime/env issues (missing libs, API key instantiation timing, port in use, RabbitMQ not running). It was faster to fix directly."

• Opus 4 insight:

  "Look, I built those tools with knowledge that I already have. When I am actually doing the task, it's easier for me to just do it rather than using the tools"

Key Insights

Model Behavior Patterns

• Tool learning resistance:

  "Nathan Lambert saying that models quickly learn to NOT use a tool during RL process if they have early failures"

• Scale vs scaffolding:

  "Noam Brown saying that scaffolding for agents will be washed away by scale [...] This was the first time I really felt what he meant first hand"

• Enforcement need:

  "having them pickup new tools at inference time needs stronger enforcement than just prompting them to do it"

AGI Asymptote Theory

• Deceleration perception:

  "The perceived deceleration in model improvements is explained above. Until the AGI line is crossed, it will be harder and harder to perceive big jumps"

• Arbitrage opportunity:

  "If that's the case, it means that in many tasks the performance of older models is almost AGI, except much cheaper and often open source"

Conclusions

• Current state:

  "For now, I think we are far from inference-time self-improving coding agents that really push the frontier"

• Practical recommendation:

  "I still think it's a great idea to use models to improve your rule-based tools. Writing ESLint rules, tests, etc is always a good investment of tokens"

• Future research direction:

  "I'd look into having the model perfect these tools and then do some sort of RL over them to really internalize them, and see if that would make a difference"

References

• Related work: Voyager - early approach to inference-time self-improvement through skill library

• OpenAI benchmarks: MLE Bench, SWE-Lancer

• Tools mentioned: claude-squad, vibe-kanban, Linear MCP

• Full results: GitHub repo

• Author context: Working on Kernel Labs

Cline: Open Source Code Agent - Research Summary

Company Overview & Product

• Cline is an open source coding agent as VS Code extension (also coming to JetBrains, NeoVim, CLI)

  "Cline's an open source coding agent. It's a VS Code extension right now, but it's coming to JetBrains and NeoVim and CLI."

• Approaching 2 million downloads, launched January 2025

• Announced $32M Series A funding

• Vision: Infrastructure layer for agents

  "Cline is the kind of infrastructure layer for agents, for all open source agents, people building on top of this like agentic infrastructure."

Core Innovation: Plan + Act Paradigm

• Pioneered two-mode system for agent interaction

  "Cline was the first to sort of come up with this concept of having two modes for the developer to engage with."

• Plan mode: Exploratory, read files, gather context, extract requirements from developer

  "in plan mode, the agents directed to be more exploratory, read more files, get more data"

• Act mode: Execute on plan, run commands, edit files with optional auto-approve

  "when they switch to act mode, that's when the agent gets this directive to look at the plan and start executing on it"

• Emerged organically from user behavior patterns observed in Discord community

Technical Philosophy: Simplicity Over Complexity

Against RAG for Coding

• Article: Why I No Longer Recommend RAG for Code

  "RAG is a mind virus"

• Critique of RAG approach:

  "the way rag works is you have to like chunk all these files across your entire repository and like chop them up in a small little piece. And then throw them into this hyper dimensional vector space, and then pull out these random chugs when you're searching for relevant code snippets. And it's like, fundamentally, it's like so schizo."

• Prefers agentic search: mimics senior engineer exploration pattern

  "you look at the folder structure, you look through the files, oh, this file imports from this other file, let's go take a look at that. And you kind of agentically explore the repository."

Fast Apply Models "Bitter Lesson'd"

• Article: Fast Apply Models Are Dead
• Fast apply: Fine-tuned small models to handle lazy code snippets from frontier models

• Problems with fast apply:

  "now instead of worrying about one model messing things up, now you have to worry about two models messing things up"

  "At like when fast apply came out, that was way higher, that was like in the 20s and the 30s. Now we're down to 4%"

• Claude Sonnet 4 achieved sub-5% diff edit failure rate, making fast apply obsolete

• Founders of fast apply companies estimate 3-month relevance window

Context Engineering Approach

Dynamic Context Management

• Provides maximum visibility into model actions: prompts, tool calls, errors

  "We try to give as much insight into what exactly the model is doing in each step in accomplishing a task."

• Uses AST (Abstract Syntax Trees) for code navigation

  "there's a tool that lets it pull in all the sort of language from a directory. So, it could be the names of classes, the names of functions"

• Incorporates open VS Code tabs as context hints

  "what tabs they have open in VS Code. That was actually in our internal kind of benchmarking that turned out to work very, very well."

Narrative Integrity

• Treats each task as story with coherent arc

  "every task and client is kind of like a story...how do we maintain that narrative integrity where every step of the way the agent can kind of predict the next token"

• Context summarization by asking model what's relevant rather than naive truncation

• To-do list tool experiment: maintains agent focus across 10x context window length

Memory Systems

• Memory Bank concept for tribal knowledge

  "how can we hold on to the tribal knowledge that these agents learn along the way that people aren't documenting or putting into rules files"

• Scratch pad approach: passive tracking of work state

• Separate rules files (cline_rules) from other tools preferred by founders

MCP (Model Context Protocol) Integration

Early Adoption & Marketplace

• Launch partner for Anthropic's MCP

• MCP Marketplace launched February 2025 with 150+ servers

  "we launched the MCP marketplace where you could actually go through and have this one-click install process"

• System prompt initially heavily focused on teaching MCP to models

Popular MCP Servers

• File System MCP
• Browser automation: Browser Tools, Playwright, Puppeteer
• Git Tools
• Context7: documentation retrieval across libraries
• Perplexity Research
• Slack, Unity, Ableton integrations

Non-Technical Use Cases

• Marketing automation: Reddit scraping → Twitter posting via MCPs

  "Nick Bauman, he uses it to connect to, you know, a Reddit MCP server, scrape content connected to an X MCP server and post tweets"

• Presentation creation using SlideDev + Limitless transcription

• Example workflow: automated PR review → Slack notification

  "pull down this PR...Pull in all that context, read the files around the diff, review it...approve it and then send a message in Slack"

MCP Monetization & Security

• 21st.dev Magic MCP: Monetizes via API keys for beautiful UI components

  "they have this library of beautiful components and they just inject relevant examples"

• Security concerns: malicious code in forks, need for version locking

• Stripe exploring unified payment layer for MCP tools
• Future vision: agents paying for tool calls autonomously via stablecoins

Business Model & Enterprise

Open Source + BYOK (Bring Your Own API Key)

• Direct connection to model providers (Anthropic, OpenAI, Bedrock, OpenRouter)

  "Right now, it's bringing an API key, essentially just whatever pre-commitment you might have to whatever inference provider"

• No margin capture on inference

  "our thesis is inference is not the business"

• Transparency in pricing and data routing builds trust

  "that level of transparency, that level of we're building the best product. We're not focused on sort of capturing margin"

Enterprise Offering

• Fortune 5 companies demanded enterprise features

  "we have hundreds of engineers using Cline within our organization and this is a massive problem for us...Please just like, let us give you money"

• Features: governance, security guardrails, usage insights, invoicing

• Self-hosted option with internal router (similar to OpenRouter architecture)
• ROI metrics: lines of code, usage statistics for internal champions

Fork Ecosystem

• 6,000+ forks of Cline
• Top 3 apps in OpenRouter usage are Cline variants
• Samsung created isolated fork mentioned in Wall Street Journal

• No regrets about open source approach

  "let them copy. We're the leaders in the space. We're kind of showing the way for the entire industry."

Model Evolution & Evaluation

• Started 10 days after Claude 3.5 Sonnet release (June 2024)

• Anthropic's model card addendum on agentic coding capabilities inspired development

  "there was this section about agentic coding and how it was so much better at this step by step accomplishing tasks"

• Focus on models' improved long-context understanding (needle in haystack)

• Claude Sonnet 4: ~4% diff edit failure rate (down from 20-30%)

Competitive Positioning

IDE Integration Matrix

• Visibility axis: How much insight into agent actions
• Autonomy axis: How automated the process is

• Cline position: High visibility, balanced autonomy for "serious engineering teams"

  "serious engineering teams where they can't really give everything over to the AI, at least not yet. And they need to have high visibility"

• Complements other tools: Cursor for inline edits, Windsurf for developer experience

  "being an extension also gives us a lot more distribution. You have to use us or somebody else."

Avoiding VS Code Fork

• Chose extension over fork to avoid maintenance burden

  "Microsoft makes it like notoriously difficult to maintain these forks"

• Benefits: broader distribution, focus on core agentic loop, compatibility with Cursor/Windsurf

Future Modalities

• Background agents (like Codex, Devin) complement interactive agents
• Parallel agents (Kanban interfaces) for experimentation

• CLI version enabling cloud deployment, GitHub actions

  "the CLI is really the form factor for these kind of fully autonomous agents"

• SDK for building agents on Cline infrastructure

Key Technical Insights

Complexity Redefinition

• Past complexity: Algorithmic challenges (now trivial for models)

• Current complexity: Architectural decisions, vision, taste

  "what we might have considered complex a few years ago, algorithmic, you know, challenges, that's pretty trivial for models today"

  "architectural decisions are a lot more fun to think about than putting together algorithms"

Course Correction Critical

• Real-time feedback more valuable than autonomous completion

  "the course correcting part is so incredibly important and in getting work done, I think much more quickly than if you were to kind of give a sort of a background agent work"

Anthropomorphization Benefits

• Named personality ("Cline" - play on CLI + editor)

• Humanization builds trust and improves results

  "the humanizing aspect of it, I think has been helpful to me personally...There's, there's kind of a, of a trust building"

  "it's actually really important, I think, to anthropomorphize agents in general, because everything they do is like a little story"

Team & Culture

• 20 people, aiming for 100 by end of year
• Hiring primarily through network: friends of friends
• Culture: "feels like we're all just like friends building something cool"
• Open source creates goodwill with constructive user feedback
• Activities: go-karting, kayaking alongside intense work

Referenced Tools & Companies

• Competitors/Alternatives: Cursor, Windsurf, Copilot, Ader, Codex, Devin (Cognition Labs), Replit, Lovable
• Related Tools: OpenRouter, Sentry, Agents-927, Kiro, Warp 2.0, Charm Crush, Augment CLI
• Technologies: VS Code, JetBrains, NeoVim, Claude models, GPT models, Gemini, DeepSeek
• Services: Stripe, GitHub, Slack, Reddit, X/Twitter, Unity, Ableton, Cloudflare Workers

OpenAI Dev Day 2025: AgentKit & Platform Strategy

Overview & Platform Vision

• OpenAI positions developers as the distribution layer for AGI benefits: > "our mission at OpenAI is to, one, build AGI...and then...just as important is to bring the benefits of that to the entire world...we really need to rely on developers, other third parties to be able to do this"
• Developer ecosystem growth: 4 million developers (up from ~3 million last year)
• ChatGPT now 5th or 6th largest website globally with 800 million weekly active users

• "Today we're going to open up ChatGPT for developers to build real apps inside of ChatGPT...with the Apps SDK, your apps can reach hundreds of millions of ChatGPT users" — Sam Altman

Major Model Releases

API Parity with Consumer Products: - GPT-5 Pro - flagship model now available via API - Sora 2 & Sora 2 Pro - video generation models released - Distilled models: - gpt-realtime-mini (70% cheaper) - gpt-audio-mini - gpt-image-1-mini (80% cheaper)

Apps SDK & MCP Integration

• Built on Model Context Protocol (MCP), first major platform to adopt it

• "OpenAI adopted [MCP] so quickly, much less to now be the first to turn it into the basis of a full app store platform"

• Technical innovations:
• React component bundling for iframe targets with custom UI components
• Live data flow (demonstrated with Coursera app allowing queries during video watching)
• OpenAI joined MCP steering committee in March 2025, with Nick Cooper as representative

• "they really treat it as an open protocol...they are not viewing it as this thing that is specific to Anthropic"

AgentKit Platform Components

Agent Builder

• Visual workflow builder with drag-and-drop interface

• "launched agent kit today, full set of solutions to build, deploy and optimize agents"

• Supports both deterministic and LLM-driven workflows
• Uses Common Expression Language (CEL) for conditional logic
• Features: user approval nodes, transform/set state capabilities, templating system
• Pre-built templates: customer support, document discovery, data enrichment, planning helper, structured data Q&A, document comparison, internal knowledge assistant

Agent SDK

• "allowing you to use [traces] in the evals product and be able to grade it...over the entirety of what it's supposed to be doing"

• Supports MCP protocol integration
• Enables code export from Agent Builder for standalone deployment
• Built-in tracing capabilities for debugging and evaluation

ChatKit

• Consumer-grade embeddable chat interface

• "ChatKit itself is like an embeddable iframe...if you are using ChatKit and we come up with new...a new model that reasons in a different way...you don't actually need to rebuild"

• Designed by team that built Stripe Checkout
• Provides "full stack" with widgets and custom UI components
• Already powers help.openai.com customer support

Connector Registry

• First-party "sync connectors" that store state for re-ranking and optimization
• Third-party MCP server support

• "we end up storing quite a bit of state...we can actually end up doing a lot more creative stuff...when you're chatting with ChatGPT"

• Tradeoffs between first-party depth vs third-party breadth discussed

Evaluation Tools

• Agent-specific eval capabilities for multi-step workflows

• "how do you even evaluate a 20 minute task correctly? And it's like, it's a really hard problem"

• Multi-model support including third-party models via OpenRouter integration
• Automated prompt optimization with LM-as-judge rubrics
• Future plans for component-level evaluation of complex traces

Developer Experience Insights

Prompt Engineering Evolution

• "two years ago people were like, oh, at some point...prompting is going to be dead...And if anything, it is like become more and more entrenched"

• Research advancing with GEPA (Databricks) and other optimization techniques

• "it is like pretty difficult for us to manage all of these different [fine-tuning] snapshots...if there is a way to...do this like zero gradient like optimization via prompts...I'm all for it"

Internal Codex Usage

• Agent Builder built in under 2 months using Codex

• "on their way to work, they're like kicking off like five Codex tasks because the bus takes 30 minutes...and it kind of helps you orient yourself for the day"

• High-quality PR reviews from Codex widely adopted internally
• Pattern shift: > "push yourself to like trust the model to do more and more...full YOLO mode, like trust it to like write the whole feature"

Infrastructure & Reliability

Service Health Dashboard

• New org-scoped SLO tracking for API integrations
• Monitors token velocity (TPM), throughput, response codes in real-time

• "We haven't had one [major outage] that bad since...We think we've got reliability in a spot where we're comfortable kind of putting this out there"

• Target: moving from 4 nines toward 5 nines availability (exponentially more work per nine)
• Serving >6 billion tokens per minute (stat already outdated at time of interview)

Strategic Partnerships

• Apple Siri integration: ChatGPT account status determines model routing (free vs Plus/Pro)
• Kakao (Korea's largest messenger app): Sign-in with ChatGPT integration
• Jony Ive and Stargate announcements happening offstage

Key Personalities

• Sherwin Wu - Head of Engineering, OpenAI Platform
• Christina Huang - Platform Experience, OpenAI
• John Schulman - Now at xAI, launched Tinker API (low-level fine-tuning library he championed at both OpenAI and Anthropic)
• Michelle Pokrass - Former API team (2024), championed "API = AGI" philosophy
• Greg Brockman - Mentioned sustainable businesses built on Custom GPTs
• Sam Altman - Delivered keynote, announced Apps SDK

References & Tools

• OpenAI Platform Docs
• OpenAI Cookbook - Sora 2 Prompting Guide
• chatkit.studio - Playground and demos
• widget.studio - Widget builder
• chatkit.world - Interactive demo site
• Ben Thompson's endorsement of Christina's demo
• Model Context Protocol (MCP) consortium

Future Directions

• Multimodal evals expansion
• Voice modality for Agent Builder
• Human-in-the-loop workflows over weeks, not just binary approvals
• Bring-your-own-key (BYOK) for public agent deployments
• Protocol standardization (responses API, agent workflows)
• Enhanced widget ecosystem potentially user-contributed

Sleep-time Compute: Beyond Inference Scaling at Test-time

Core Concept

• Sleep-time compute allows models to "think" offline about contexts before queries are presented, reducing test-time compute requirements by ~5× on benchmark tasks

"by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time"

• The approach works by processing context c during idle time to create an enhanced representation c', which is then used at test-time: S(c) → c', followed by Tb(q, c') → a

"In practice, this is achieved by prompting the model to generate a new context consisting of inferences about the existing context, which may be potentially useful for answering test-time queries"

Key Results

• Performance improvements: Sleep-time compute reduces test-time compute needed to achieve same accuracy by ~5× on Stateful GSM-Symbolic and Stateful AIME

"Sleep-time compute produces a pareto improvement in the test-time compute vs. accuracy curve, reducing the test-time compute needed to achieve the same accuracy by ∼ 5×"

• Scaling benefits: By scaling up sleep-time compute, accuracy increases by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME

• Cost amortization: When multiple queries share the same context, average cost per query decreases by 2.5×

"By amortizing sleep-time compute across related queries about the same context using Multi-Query GSM-Symbolic, we can decrease the average cost per query by 2.5×"

Datasets Introduced

• Stateful GSM-Symbolic: Modified from GSM-Symbolic (P1: 5000 examples, P2: 2500 examples) by splitting problems into context and question

"We introduce two datasets to study applying sleep-time compute in stateful settings, Stateful GSM-Symbolic, and Stateful AIME – by splitting the existing problems in these datasets into a context and a question"

• Stateful AIME: Contains 60 questions from AIME 2024 and 2025, split into context and query components

• Multi-Query GSM-Symbolic: Extends GSM-Symbolic with multiple related queries per context (P1: 12,043 questions, 1,095 contexts; P2: 5,497 questions, 500 contexts)

• SWE-Features: Software engineering benchmark for multi-file feature implementation tasks (33 examples from Aider-AI/aider and ComfyUI repositories)

Models Evaluated

• Non-reasoning models: GPT-4o-mini and GPT-4o on GSM-Symbolic tasks

• Reasoning models: OpenAI's o1, o3-mini, Anthropic's Claude Sonnet 3.7 Extended Thinking, and DeepSeek-R1 on AIME tasks

• Test-time compute scaled both sequentially (varying verbosity/reasoning effort) and in parallel (pass@k sampling)

Effectiveness Analysis

• Query predictability correlation: Sleep-time compute is most effective when queries are predictable from context

"sleep-time compute is more effective in settings where the query is more easily predictable from the context"

• Predictability measured using log-probability of question given context under Llama2-70B base model

• Accuracy gap between sleep-time and test-time compute widens for more predictable questions (binned analysis across 5 quantiles)

Implementation Details

• Sleep-time compute implemented via function calling with two functions: - rethink_memory: Takes new string input and replaces current context - finish_rethinking: Terminates sleep-time compute process

• Models allowed up to 10 calls to rethink_memory function

• Cost modeling assumes test-time tokens are 10× more expensive than sleep-time tokens (t=10) due to latency optimization

"Since at test-time, there are strict latency constraints, and latency optimized inference can be roughly 10× more expensive, we model the total cost of inference between both sleep-time and test-time, by up-weighing the cost of test-time tokens"

Comparison to Baselines

• Pass@k parallel scaling: Sleep-time compute consistently outperforms pass@k at same test-time token budget

"sleep-time compute consistently outperforms pass@k parallel scaling at the same test-time token budget, demonstrating that sleep-time compute can be a more effective way to scale inference-time compute than standard parallel test-time scaling"

• Context-only baseline: Sleep-time compute significantly outperforms models that only receive context and must guess the question, demonstrating questions are not trivially predictable

SWE-Features Case Study

• At lower test-time budgets, sleep-time compute achieves ~1.5× reduction in test-time tokens with higher F1 scores

• At higher budgets, standard test-time compute performs better, with higher precision but comparable recall

• Hypothesis: sleep-time compute explores more files, leading to editing more files and slightly lower precision

Related Work & Context

• Builds on recent test-time scaling approaches: sequential (OpenAI o1, DeepSeek-R1) and parallel (pass@k, best-of-N)

• Connection to speculative decoding (Leviathan et al., 2023): Both speculate on user queries, but sleep-time compute uses generated tokens as input regardless of actual query

• Connection to pre-computation in systems: Similar to memory caches (Smith, 1982) and data cubes for OLAP workloads (Gray et al., 1997)

• Resembles representation learning but operates in natural language space rather than parameter/activation space

Limitations & Future Directions

• Sleep-time compute less effective when queries are unpredictable or unrelated to context

• Current approach assumes simple two-phase interaction (sleep-time and test-time), but real-world scenarios involve multiple interaction rounds

• Future work: Optimal allocation of compute between sleep-time and test-time based on query predictability

• Potential application to synthetic data generation at scale for pretraining

Authors & Affiliation

Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. Gonzalez (Letta & UC Berkeley)

Code and data: https://github.com/letta-ai/sleep-time-compute

Scaling Context Requires Rethinking Attention

Core Thesis

• Neither transformers nor sub-quadratic architectures are well-suited for long-context training

  "the cost of processing the context is too expensive in the former, too inexpensive in the latter"

• Power attention introduced as solution: A linear-cost sequence modeling architecture with independently adjustable state size > "an architectural layer for linear-cost sequence modeling whose state size can be adjusted independently of parameters"

Three Requirements for Long-Context Architectures

1. Balanced Weight-to-State Ratio (WSFR)

• Weight-state FLOP ratio should approach 1:1 for compute-optimal models

  "for compute-optimal models, the WSFR should be somewhat close to 1:1"

• Exponential attention becomes unbalanced at long contexts

• At 65,536 context: WSFR is 1:8

• At 1,000,000 context: WSFR is 1:125

  "exponential attention is balanced for intermediate context lengths, but unbalanced for long context lengths, where it does far more state FLOPs than weight FLOPs"

• Linear attention remains unbalanced at all context lengths

• WSFR stays at 30:1 regardless of context length

  "Linear attention...is unbalanced at all context lengths in the opposite direction: far more weight FLOPs than state FLOPs"

2. Hardware-Aware Implementation

• Must admit efficient implementation on tensor cores
• Power attention achieves 8.6x faster throughput than Flash Attention at 64k context (head size 32)
• 3.3x speedup at head size 64

3. Strong In-Context Learning (ICL)

• Large state size improves ICL performance

  "state scaling improves performance"

• Windowed attention fails ICL beyond window size

  "no in-context learning occurs beyond 100 tokens for window-32 attention"

• Linear attention maintains ICL across entire sequence

  "linear attention...demonstrate consistent in-context learning across the entire sequence"

Power Attention Technical Details

Mathematical Foundation

• Power attention formula: Uses p-th power instead of exponential

  "attnᵖₚₒw(Q, K, V)ᵢ = Σⱼ₌₁ⁱ (QᵢᵀKⱼ)ᵖVⱼ"

• Symmetric power expansion (SPOW) reduces state size vs tensor power (TPOW)

• At p=2, d=64: SPOW uses 2,080 dimensions vs TPOW's 4,096 (49% savings)
• At p=4, d=64: 95% size reduction

  "SPOWₚ is a state expansion that increases the state size by a factor of (ᵈ⁺ᵖ⁻¹ₚ)/d without introducing any parameters"

Implementation Innovation

• Fused expand-MMA kernel: Expands tiles on-the-fly during matrix multiplication

  "a matrix multiplication where the tiles of one operand are expanded on-the-fly"

• Tiled symmetric power expansion (TSPOW): Interpolates between TPOW and SPOW

• Provides GPU-friendly structure while reducing data duplication

• Optimal tile size: d-tile = 8 for p=2, d-tile = 4 for p=3

• Chunked form enables practical efficiency

  "The chunked form interpolates between the recurrent form and the attention form, capturing benefits of both"

• Cost: O(tDv + tcd) where c is chunk size

Experimental Results

In-Context Learning Performance

• Power attention dominates windowed attention at equal state sizes across all context lengths
• All scaling axes improve ICL: gradient updates, batch size, parameter count, context length

  "In all cases, the ICL curve becomes steeper as we scale the respective axis"

Long-Context Training (65,536 tokens)

• Power attention (p=2) outperforms both exponential and linear attention in loss-per-FLOP
• RWKV with power attention shows near-zero ICL benefit beyond 2,000 tokens
• Power attention enables RWKV to ICL "nearly as well as exponential attention"

Compute-Optimal Under Latency Constraints

• When inference latency constrains parameter count and state size:
• Window-1k attention: loss 1.638
• Standard attention: loss 1.631
• Power attention (p=2): loss 1.613 (best)

Dataset and Experimental Setup

LongCrawl64

• 6.66M documents, each 65,536 tokens (435B total tokens)
• Sourced from Common Crawl, filtered for long sequences
• Critical for ICL research

  "Most sequences in OpenWebText have length less than 1k"

Architectures Tested

• Base architectures: GPT-2, RWKV (RWKV7), GLA, RetNet
• Attention variants: Exponential, linear, windowed, power (p=2)
• Training: LongCrawl64, AdamW, bf16, learning rate 3e-4 with warmup and cosine decay

Key Limitations and Future Work

Current Limitations

1. Experiments limited to natural language NLL - no other domains/modalities tested
2. Compute-optimal context grows slowly in natural language

   "autoregressive prediction of natural language is largely dominated by short-context dependencies"

3. p=2 only - normalization requires positive inner products (even powers only)
4. Triton implementation - not yet optimized to CUDA level

Future Directions

• Explore domains with long-term dependencies: chain-of-thought reasoning, audio, video
• Scaling laws research for state size, context size, and ICL
• CUDA implementation for further speedups beyond current Triton kernels
• Alternative normalization to support odd powers
• Comprehensive comparison to hybrid models, sparse attention, MQA, latent attention

Key References and Tools

Implementations

• Open-source kernels: https://github.com/m-a-n-i-f-e-s-t/power-attention
• FLA codebase: Flash Linear Attention [Yang and Zhang, 2024]
• PyTorch framework [Paszke et al., 2019]

Related Techniques

• Flash Attention [Dao, 2023]: Operator fusion to avoid materializing attention matrix
• Linear attention [Katharopoulos et al., 2020]: Enables recurrent formulation
• Gating [Lin et al., 2025]: Learned mechanism to avoid attending to old data
• Sliding window attention [Child et al., 2019]: Truncates KV cache

Key Papers

• Transformers [Vaswani et al., 2023]
• Mamba [Gu and Dao, 2024]: Modern RNN architecture
• RWKV [Peng et al., 2023]: Reinventing RNNs for transformer era
• Scaling laws [Kaplan et al., 2020]

Technical Contributions

1. Framework for evaluating long-context architectures (balance, efficiency, ICL)
2. Power attention architecture with parameter-free state size adjustment
3. Symmetric power expansion theory and implementation
4. Hardware-efficient kernels with operation fusion
5. Empirical validation on 435B token dataset

The Prompt Report: A Systematic Survey of Prompting Techniques

Overview & Scope

• Comprehensive taxonomy: "We establish a structured understanding of prompt engineering by assembling a taxonomy of prompting techniques and analyzing their applications. We present a detailed vocabulary of 33 vocabulary terms, a taxonomy of 58 LLM prompting techniques, and 40 techniques for other modalities."

• Scope limitation: "We limit our study to focus on prefix prompts rather than cloze prompts, because modern LLM transformer architectures widely employ prefix prompts"

• Focus on hard prompts: "Additionally, we refined our focus to hard (discrete) prompts rather than soft (continuous) prompts and leave out papers that make use of techniques using gradient-based updates (i.e. fine-tuning). Hard prompts contain only tokens (vectors) that correspond to words in the model's vocabulary"

Key Definitions

Prompt & Prompting

• Prompt definition: "A prompt is an input to a Generative AI model, that is used to guide its output"

• Prompt template: "A prompt template is a function that contains one or more variables which will be replaced by some media (usually text) to create a prompt"

• Prompting: "Prompting is the process of providing a prompt to a GenAI, which then generates a response"

Prompt Engineering

• Consolidated definition: "Prompt engineering is the iterative process of developing a prompt by modifying or changing the prompting technique that you are using"

• Process description: "The Prompt Engineering Process consists of three repeated steps 1) performing inference on a dataset 2) evaluating performance and 3) modifying the prompt template"

Core Prompt Components

Essential Elements

• Directive: "Many prompts issue a directive in the form of an instruction or question. This is the core intent of the prompt"

• Examples/Exemplars: "Examples, also known as exemplars or shots, act as demonstrations that guide the GenAI to accomplish a task"

• Output formatting: "It is often desirable for the GenAI to output information in certain formats, for example, CSV, Markdown, XML, or even custom formats"

• Style instructions: "Style instructions are a type of output formatting used to modify the output stylistically rather than structurally"

• Role/Persona: "A Role, also known as a persona, is a frequently discussed component that can improve writing and style text"

Systematic Review Methodology

PRISMA Process

• Approach: "We conducted a machine-assisted systematic review grounded in the PRISMA process to identify 58 different text-based prompting techniques"

• Data sources: "Our main data sources were arXiv, Semantic Scholar, and ACL. We query these databases with a list of 44 keywords narrowly related to prompting and prompt engineering"

• Pipeline: "We retrieve papers from arXiv based on a simple set of keywords and boolean rules. Then, human annotators label a sample of 1,661 articles"

• Inter-rater reliability: "A set of 300 articles are reviewed independently by two annotators, with 92% agreement (Krippendorff's α = Cohen's κ = 81%)"

• Final dataset: "The combined human and LLM annotations generate a final set of 1,565 papers"

Major Technique Categories

In-Context Learning (ICL)

• Definition: "ICL refers to the ability of GenAIs to learn skills and tasks by providing them with exemplars and or relevant instructions within the prompt, without the need for weight updates/retraining"

• Few-Shot Prompting: "Brown et al. (2020) is the paradigm seen in Figure 2.4, where the GenAI learns to complete a task with only a few examples (exemplars)"

Design Decisions for Few-Shot Prompting

• Exemplar quantity: "Increasing the quantity of exemplars in the prompt generally improves model performance, particularly in larger models. However, in some cases, the benefits may diminish beyond 20 exemplars"

• Exemplar ordering: "The order of exemplars affects model behavior. On some tasks, exemplar order can cause accuracy to vary from sub-50% to 90%+"

• Label distribution impact: "As in traditional supervised machine learning, the distribution of exemplar labels in the prompt affects behavior"

• Label quality: "Despite the general benefit of multiple exemplars, the necessity of strictly valid demonstrations is unclear. Some work suggests that the accuracy of labels is irrelevant—providing models with exemplars with incorrect labels may not negatively diminish performance"

• Exemplar format: "The formatting of exemplars also affects performance. One of the most common formats is 'Q: {input}, A: {label}', but the optimal format may vary across tasks"

• Exemplar similarity: "Selecting exemplars that are similar to the test sample is generally beneficial for performance. However, in some cases, selecting more diverse exemplars can improve performance"

Few-Shot Techniques

• K-Nearest Neighbor (KNN): "Liu et al. (2021) is part of a family of algorithms that selects exemplars similar to test samples to boost performance"

• Vote-K: "Su et al. (2022) is another method to select similar exemplars to the test sample... Vote-K also ensures that newly added exemplars are sufficiently different than existing ones to increase diversity"

• Self-Generated In-Context Learning (SG-ICL): "Kim et al. (2022) leverages a GenAI to automatically generate exemplars. While better than zero-shot scenarios when training data is unavailable, the generated samples are not as effective as actual data"

• Prompt Mining: "Jiang et al. (2020) is the process of discovering optimal 'middle words' in prompts through large corpus analysis"

Zero-Shot Techniques

• Role Prompting: "Wang et al. (2023j); Zheng et al. (2023d), also known as persona prompting, assigns a specific role to the GenAI in the prompt"

• Style Prompting: "Lu et al. (2023a) involves specifying the desired style, tone, or genre in the prompt to shape the output"

• Emotion Prompting: "Li et al. (2023a) incorporates phrases of psychological relevance to humans (e.g., 'This is important to my career') into the prompt, which may lead to improved LLM performance"

• System 2 Attention (S2A): "Weston and Sukhbaatar (2023) first asks an LLM to rewrite the prompt and remove any information unrelated to the question therein"

• Rephrase and Respond (RaR): "Deng et al. (2023) instructs the LLM to rephrase and expand the question before generating the final answer"

• Re-reading (RE2): "Xu et al. (2023) adds the phrase 'Read the question again:' to the prompt in addition to repeating the question"

• Self-Ask: "Press et al. (2022) prompts LLMs to first decide if they need to ask follow up questions for a given prompt"

Thought Generation

• Chain-of-Thought (CoT): "Wei et al. (2022b) leverages few-shot prompting to encourage the LLM to express its thought process before delivering its final answer"

• Zero-Shot-CoT: "The most straightforward version of CoT contains zero exemplars. It involves appending a thought inducing phrase like 'Let's think step by step.' to the prompt"

• Step-Back Prompting: "Zheng et al. (2023c) is a modification of CoT where the LLM is first asked a generic, high-level question about relevant concepts or facts before delving into reasoning"

• Thread-of-Thought (ThoT): "Zhou et al. (2023) consists of an improved thought inducer for CoT reasoning. Instead of 'Let's think step by step,' it uses 'Walk me through this context in manageable parts step by step, summarizing and analyzing as we go.'"

• Tabular Chain-of-Thought (Tab-CoT): "Jin and Lu (2023) consists of a Zero-Shot CoT prompt that makes the LLM output reasoning as a markdown table"

Few-Shot CoT Variants

• Contrastive CoT: "Chia et al. (2023) adds both exemplars with incorrect and correct explanations to the CoT prompt in order to show the LLM how not to reason"

• Complexity-based Prompting: "Fu et al. (2023b) involves two major modifications to CoT. First, it selects complex examples for annotation and inclusion in the prompt... Second, during inference, it samples multiple reasoning chains"

• Active Prompting: "Diao et al. (2023) starts with some training questions/exemplars, asks the LLM to solve them, then calculates uncertainty (disagreement in this case) and asks human annotators to rewrite the exemplars with highest uncertainty"

• Memory-of-Thought: "Li and Qiu (2023b) leverage unlabeled training exemplars to build Few-Shot CoT prompts at test time"

• Automatic Chain-of-Thought (Auto-CoT): "Zhang et al. (2022b) uses Wei et al. (2022b)'s Zero-Shot prompt to automatically generate chains of thought. These are then used to build a Few-Shot CoT prompt"

Decomposition

• Least-to-Most Prompting: "Zhou et al. (2022a) starts by prompting a LLM to break a given problem into sub-problems without solving them. Then, it solves them sequentially, appending model responses to the prompt each time"

• Decomposed Prompting (DECOMP): "Khot et al. (2022) Few-Shot prompts a LLM to show it how to use certain functions. These might include things like string splitting or internet searching"

• Plan-and-Solve Prompting: "Wang et al. (2023f) consists of an improved Zero-Shot CoT prompt, 'Let's first understand the problem and devise a plan to solve it. Then, let's carry out the plan and solve the problem step by step'"

• Tree-of-Thought (ToT): "Yao et al. (2023b), also known as Tree of Thoughts, creates a tree-like search problem by starting with an initial problem then generating multiple possible steps in the form of thoughts"

• Program-of-Thoughts: "Chen et al. (2023d) uses LLMs like Codex to generate programming code as reasoning steps. A code interpreter executes these steps to obtain the final answer"

• Skeleton-of-Thought: "Ning et al. (2023) focuses on accelerating answer speed through parallelization. Given a problem, it prompts an LLM to create a skeleton of the answer"

Ensembling

• Demonstration Ensembling (DENSE): "Khalifa et al. (2023) creates multiple few-shot prompts, each containing a distinct subset of exemplars from the training set. Next, it aggregates over their outputs"

• Self-Consistency: "Wang et al. (2022) is based on the intuition that multiple different reasoning paths can lead to the same answer. This method first prompts the LLM multiple times to perform CoT, crucially with a non-zero temperature"

• Universal Self-Consistency: "Chen et al. (2023e) is similar to Self-Consistency except that rather than selecting the majority response by programmatically counting how often it occurs, it inserts all outputs into a prompt template"

• DiVeRSe: "Li et al. (2023i) creates multiple prompts for a given problem then performs Self-Consistency for each, generating multiple reasoning paths"

• Prompt Paraphrasing: "Jiang et al. (2020) transforms an original prompt by changing some of the wording, while still maintaining the overall meaning"

Self-Criticism

• Self-Calibration: "Kadavath et al. (2022) first prompts an LLM to answer a question. Then, it builds a new prompt that includes the question, the LLM's answer, and an additional instruction asking whether the answer is correct"

• Self-Refine: "Madaan et al. (2023) is an iterative framework where, given an initial answer from the LLM, it prompts the same LLM to provide feedback on the answer, and then prompts the LLM to improve the answer based on the feedback"

• Self-Verification: "Weng et al. (2022) generates multiple candidate solutions with Chain-of-Thought (CoT). It then scores each solution by masking certain parts of the original question"

• Chain-of-Verification (COVE): "Dhuliawala et al. (2023) first uses an LLM to generate an answer to a given question. Then, it creates a list of related questions that would help verify the correctness of the answer"

Prompt Engineering Automation

Meta Prompting

• Definition: "Meta Prompting is the process of prompting a LLM to generate or improve a prompt or prompt template"

Automated Techniques

• AutoPrompt: "Shin et al. (2020b) uses a frozen LLM as well as a prompt template that includes some 'trigger tokens', whose values are updated via backpropagation at training time"

• Automatic Prompt Engineer (APE): "Zhou et al. (2022b) uses a set of exemplars to generate a Zero-Shot instruction prompt. It generates multiple possible prompts, scores them, then creates variations of the best ones"

• Gradientfree Instructional Prompt Search (GrIPS): "Prasad et al. (2023) is similar to APE, but uses a more complex set of operations including deletion, addition, swapping, and paraphrasing"

• RLPrompt: "Deng et al. (2022) uses a frozen LLM with an unfrozen module added. It uses this LLM to generate prompt templates, scores the templates on a dataset, and updates the unfrozen module using Soft Q-Learning"

Answer Engineering

Core Concept

• Definition: "Answer engineering is the iterative process of developing or selecting among algorithms that extract precise answers from LLM outputs"

Three Design Decisions

• Answer Shape: "The shape of an answer is its physical format. For example, it could be a token, span of tokens, or even an image or video"

• Answer Space: "The space of an answer is the domain of values that its structure may contain. This may simply be the space of all tokens, or in a binary labeling task, could just be two possible tokens"

• Answer Extractor: "In cases where it is impossible to entirely control the answer space... a rule can be defined to extract the final answer. This rule is often a simple function (e.g. a regular expression)"

Extraction Methods

• Verbalizer: "Often used in labeling tasks, a verbalizer maps a token, span, or other type of output to a label and vice-versa (injective)"

• Regex: "Regexes are often used to extract answers. They are usually used to search for the first instance of a label"

• Separate LLM: "Sometimes outputs are so complicated that regexes won't work consistently. In this case, it can be useful to have a separate LLM evaluate the output and extract an answer"

Multilingual Prompting

Core Challenges

• Performance disparity: "State-of-the-art GenAIs have often been predominately trained with English dataset, leading to a notable disparity in the output quality in languages other than English, particularly low-resource languages"

Key Techniques

• Translate First Prompting: "Shi et al. (2022) is perhaps the simplest strategy and first translates non-English input examples into English"

• Cross-Lingual Thought (XLT): "Huang et al. (2023a) utilizes a prompt template composed of six separate instructions, including role assignment, cross-lingual thinking, and CoT"

• Cross-Lingual Self Consistent Prompting (CLSP): "Qin et al. (2023a) introduces an ensemble technique that constructs reasoning paths in different languages to answer the same question"

Prompt Language Selection

• English advantage: "Constructing the prompt template in English is often more effective than in the task language for multilingual tasks. This is likely due to the predominance of English data during LLM pre-training"

• Native language rationale: "In contrast, many multilingual prompting benchmarks such as BUFFET or LongBench use task language prompts for language-specific use cases"

Machine Translation Techniques

• Multi-Aspect Prompting and Selection (MAPS): "He et al. (2023b) mimics the human translation process, which involves multiple preparatory steps to ensure high-quality output"

• Chain-of-Dictionary (CoD): "Lu et al. (2023b) first extracts words from the source phrase, then makes a list of their meanings in multiple languages, automatically via retrieval from a dictionary"

• Interactive-Chain-Prompting (ICP): "Pilault et al. (2023) deals with potential ambiguities in translation by first asking the GenAI to generate sub-questions about any ambiguities in the phrase to be translated"

Multimodal Prompting

Image Prompting

• Prompt Modifiers: "are simply words appended to a prompt to change the resultant image. Components such as Medium (e.g. 'on canvas') or Lighting (e.g. 'a well lit scene') are often used"

• Negative Prompting: "allows users to numerically weight certain terms in the prompt so that the model considers them more/less heavily than others"

Multimodal ICL

• Paired-Image Prompting: "shows the model two images: one before and one after some transformation. Then, present the model with a new image for which it will perform the demonstrated conversion"

• Image-as-Text Prompting: "Hakimov and Schlangen (2023) generates a textual description of an image. This allows for the easy inclusion of the image (or multiple images) in a text-based prompt"

Multimodal CoT

• Duty Distinct Chain-of-Thought (DDCoT): "Zheng et al. (2023b) extends Least-to-Most prompting to the multimodal setting, creating subquestions, then solving them and combining the answers"

• Chain-of-Images (CoI): "Meng et al. (2023) is a multimodal extension of Chain-of-Thought prompting, that generates images as part of its thought process"

Other Modalities

• Audio: "Experiments with audio ICL have generated mixed results, with some open source audio models failing to perform ICL. However, other results do show an ICL ability in audio models"

• Video: "Prompting has also been extended to the video modality, for use in text-to-video generation, video editing, and video-to-text generation"

• 3D: "Prompting can also be used in 3D modalities, for example in 3D object synthesis, 3D surface texturing, and 4D scene generation"

Agents

Definition

• Agent concept: "In the context of GenAI, we define agents to be GenAI systems that serve a user's goals via actions that engage with systems outside the GenAI itself"

Tool Use Agents

• Modular Reasoning, Knowledge, and Language (MRKL) System: "Karpas et al. (2022) is one of the simplest formulations of an agent. It contains a LLM router providing access to multiple tools"

• Self-Correcting with Tool-Interactive Critiquing (CRITIC): "Gou et al. (2024a) first generates a response to the prompt, with no external calls. Then, the same LLM criticizes this response for possible errors"

Code-Generation Agents

• Program-aided Language Model (PAL): "Gao et al. (2023b) translates a problem directly into code, which is sent to a Python interpreter to generate an answer"

• Tool-Integrated Reasoning Agent (ToRA): "Gou et al. (2024b) is similar to PAL, but instead of a single code generation step, it interleaves code and reasoning steps for as long as necessary"

Observation-Based Agents

• Reasoning and Acting (ReAct): "Yao et al. (2022) generates a thought, takes an action, and receives an observation (and repeats this process) when given a problem to solve"

• Reflexion: "Shinn et al. (2023) builds on ReAct, adding a layer of introspection. It obtains a trajectory of actions and observations, then is given an evaluation of success/failure"

Lifelong Learning

• Voyager: "Wang et al. (2023a) is composed of three parts. First, it proposes tasks for itself to complete in order to learn more about the world. Second, it generates code to execute these actions. Finally, it saves these actions to be retrieved later"

• Ghost in the Minecraft (GITM): "Zhu et al. (2023) starts with an arbitrary goal, breaks it down into subgoals recursively, then iteratively plans and executes actions by producing structured text"

Retrieval Augmented Generation (RAG)

• Core concept: "RAG is a paradigm in which information is retrieved from an external source and inserted into the prompt. This can enhance performance in knowledge intensive tasks"

• Verify-and-Edit: "Zhao et al. (2023a) improves on self-consistency by generating multiple chains-of-thought, then selecting some to be edited. They do this by retrieving relevant (external) information"

• Interleaved Retrieval guided by Chain-of-Thought (IRCoT): "Trivedi et al. (2023) is a technique for multi-hop question answering that interleaves CoT and retrieval"

Evaluation

Prompting Techniques for Evaluation

• In-Context Learning: "is frequently used in evaluation prompts, much in the same way it is used in other applications"

• Role-based Evaluation: "is a useful technique for improving and diversifying evaluations. By creating prompts with the same instructions for evaluation, but different roles, it is possible to effectively generate diverse evaluations"

• Chain-of-Thought: "prompting can further improve evaluation performance"

• Model-Generated Guidelines: "Liu et al. (2023d, h) prompt an LLM to generate guidelines for evaluation. This reduces the insufficient prompting problem arising from ill-defined scoring guidelines"

Output Formats

• Styling: "Formatting the LLM's response using XML or JSON styling has also been shown to improve the accuracy of the judgment generated by the evaluator"

• Linear Scale: "A very simple output format is a linear scale (e.g. 1-5). Many works use ratings of 1-10, 1-5, or even 0-1"

• Binary Score: "Prompting the model to generate binary responses like Yes or No and True or False is another frequently used output format"

• Likert Scale: "Prompting the GenAI to make use of a Likert Scale can give it a better understanding of the meaning of the scale"

Evaluation Frameworks

• LLM-EVAL: "Lin and Chen (2023) is one of the simplest evaluation frameworks. It uses a single prompt that contains a schema of variables to evaluate"

• G-EVAL: "Liu et al. (2023d) is similar to LLM-EVAL, but includes an AutoCoT steps in the prompt itself"

• ChatEval: "Chan et al. (2024) uses a multi-agent debate framework with each agent having a separate role"

Other Methodologies

• Batch Prompting: "For improving compute and cost efficiency, some works employ batch prompting for evaluation where multiple instances are evaluated at once"

• Pairwise Evaluation: "Chen et al. (2023g) find that directly comparing the quality of two texts may lead to suboptimal results and that explicitly asking LLM to generate a score for individual summaries is the most effective"

Security & Safety

Prompt Hacking

• Definition: "Prompt hacking refers to a class of attacks which manipulate the prompt in order to attack a GenAI"

• Prompt Injection: "is the process of overriding original developer instructions in the prompt with user input"

• Jailbreaking: "is the process of getting a GenAI model to do or say unintended things through prompting"

Security Risks

• Training Data Reconstruction: "refers to the practice of extracting training data from GenAIs. A straightforward example of this is Nasr et al. (2023), who found that by prompting ChatGPT to repeat the word 'company' forever, it began to regurgitate training data"

• Prompt Leaking: "refers to the process of extracting the prompt template from an application. Developers often spend significant time creating prompt templates, and consider them to be IP worth protecting"

• Package Hallucination: "occurs when LLM-generated code attempts to import packages that do not exist. After discovering what package names are frequently hallucinated by LLMs, hackers could create those packages, but with malicious code"

Defense Mechanisms

• Prompt-based Defenses: "Multiple prompt-based defenses have been proposed, in which instructions are included in the prompt to avoid prompt injection. However, Schulhoff et al. (2023) ran a study with hundreds of thousands of malicious prompts and found that no prompt-based defense is fully secure"

• Detectors: "are tools designed to detect malicious inputs and prevent prompt hacking. Many companies have built such detectors, which are often built using fine-tuned models trained on malicious prompts"

• Guardrails: "are rules and frameworks for guiding GenAI outputs. Guardrails often make use of detectors, but not always. Guardrails are more concerned with the general dialogue flow in an application"

Alignment Issues

Prompt Sensitivity

• Small changes impact: "Several works show that LLMs are highly sensitive to the input prompt, i.e., even subtle changes to a prompt such as exemplar order can result in vastly different outputs"

• Task format variation: "describes different ways to prompt an LLM to execute the same task... Zhao et al. (2021b) show that these minor changes can alter the accuracy of GPT-3 by up to 30%"

• Prompt Drift: "Chen et al. (2023b) occurs when the model behind an API changes over time, so the same prompt may produce different results on the updated model"

Calibration Issues

• Overconfidence: "LLMs are often overconfident in their answers, especially when prompted to express their own confidence in words, which may lead to user overreliance on model outputs"

• Sycophancy: "refers to the concept that LLMs will often express agreement with the user, even when that view contradicts the model's own initial output"

Bias & Fairness

• Vanilla Prompting: "Si et al. (2023b) simply consists of an instruction in the prompt that tells the LLM to be unbiased. This technique has also been referred to as moral self-correction"

• Cultural Awareness: "Yao et al. (2023a) can be injected into prompts to help LLMs with cultural adaptation"

• AttrPrompt: "Yu et al. (2023) is a prompting technique designed to avoid producing text biased towards certain attributes when generating synthetic data"

Ambiguity Handling

• Ambiguous Demonstrations: "Gao et al. (2023a) are examples that have an ambiguous label set. Including them in a prompt can increase ICL performance"

• Question Clarification: "Rao and Daumé III (2019) allows the LLM to identify ambiguous questions and generate clarifying questions to pose to the user"

Benchmarking Results

MMLU Evaluation

• Performance trends: "Performance generally improved as techniques grew more complex. However, Zero-Shot-CoT dropped precipitously from Zero-Shot. Although it had a wide spread, for all variants, Zero-Shot performed better"

• Best performer: "Few-Shot CoT performs the best, and unexplained performance drops from certain techniques need further research"

• Self-Consistency impact: "Both cases of Self-Consistency, naturally had lower spread since they repeated a single technique, but it only improved accuracy for Zero-Shot prompts"

Case Study: Suicide Crisis Detection

• Problem domain: "Our illustrative problem involves detection of signal that is predictive of crisis-level suicide risk in text written by a potentially suicidal individual"

• Target construct: "We focus here on the most important predictive factor in Suicide Crisis Syndrome assessments, referred to in the literature as either frantic hopelessness or entrapment"

• Dataset: "Two coders trained on the recognition of the factors in Suicide Crisis Syndrome coded a set of 221 posts for presence or absence of entrapment, achieving solid inter-coder reliability (Krippendorff's alpha = 0.72)"

Prompt Engineering Process

• Development effort: "The exercise proceeded through 47 recorded development steps, cumulatively about 20 hours of work. From a cold start with 0% performance, performance was boosted to an F1 of 0.53"

• Best manual approach: "10-Shot AutoDiCoT prompt includes 15 exemplars (without CoT reasoning) and one bootstrapped reasoning demonstration"

• DSPy comparison: "The best resulting prompt... achieves 0.548 F1 (and 0.385 / 0.952 precision / recall) on the test set, without making any use of the professor's email nor the incorrect instruction about the explicitness of entrapment"

Key Takeaways

• Sensitivity to details: "prompt engineering is fundamentally different from other ways of getting a computer to behave the way you want it to: these systems are being cajoled, not programmed, and... can be incredibly sensitive to specific details in prompts without there being any obvious reason those details should matter"

• Domain expertise crucial: "the third and most important take-away is that prompt engineering should involve engagement between the prompt engineer, who has expertise in how to coax LLMs to behave in desired ways, and domain experts, who understand what those desired ways are and why"

• Automation value: "Ultimately we found that there was significant promise in an automated method for exploring the prompting space, but also that combining that automation with human prompt engineering/revision was the most successful approach"

Most-Used Techniques & Models

Popular Techniques (by citations)

• Top techniques: "The prevalence of citations for Few-Shot and Chain-of-Thought prompting is unsurprising and helps to establish a baseline for understanding the prevalence of other techniques"

Popular Models (by citations in dataset)

• Top models cited include: GPT-3, GPT-4, ChatGPT, PaLM, LLaMA families

Popular Benchmarks

• Top datasets: MMLU, GSM8K, various arithmetic and commonsense reasoning benchmarks

Future Directions & Recommendations

For Beginners

• Start simple: "To those just beginning in prompt engineering, our recommendations resemble what one would recommend in any machine learning setting: understand the problem you are trying to solve (rather than just focusing on input/output and benchmark scores)"

• Stay skeptical: "It is better to start with simpler approaches first, and to remain skeptical of claims about method performance"

For Practitioners

• Contextual understanding: "To those already engaged in prompt engineering, we hope that our taxonomy will shed light on the relationships between existing techniques"

For Researchers

• Situate new work: "To those developing new techniques, we encourage situating new methods within our taxonomy, as well as including ecologically valid case studies and illustrations of those techniques"

Key References & Tools

Foundational Papers

• Brown et al. (2020) - GPT-3 and Few-Shot Learning
• Wei et al. (2022b) - Chain-of-Thought Prompting
• Kojima et al. (2022) - Zero-Shot Chain-of-Thought
• Wang et al. (2022) - Self-Consistency
• Zhou et al. (2022a) - Least-to-Most Prompting
• Yao et al. (2023b) - Tree-of-Thought
• Liu et al. (2023b) - Pre-training Survey

Agent Frameworks

• Karpas et al. (2022) - MRKL Systems
• Yao et al. (2022) - ReAct
• Shinn et al. (2023) - Reflexion
• Wang et al. (2023a) - Voyager

Tools & Platforms

• LearnPrompting.org - Maintained terminology and techniques list
• HuggingFace Dataset - Full paper corpus
• GitHub Repository - Code and resources

Evaluation & Safety

• Schulhoff et al. (2023) - HackAPrompt competition on prompt injection
• Liu et al. (2023d) - G-EVAL framework
• Chan et al. (2024) - ChatEval multi-agent debate

Multilingual & Multimodal

• Shi et al. (2022) - Multilingual Chain-of-Thought
• Zhang et al. (2023d) - Multimodal Chain-of-Thought
• Asai et al. (2023) - BUFFET multilingual benchmark

Automated Prompt Engineering

• Zhou et al. (2022b) - APE (Automatic Prompt Engineer)
• Khattab et al. (2023) - DSPy framework
• Deng et al. (2022) - RLPrompt

Dataset & Methodology Details

Dataset Composition

• Final corpus: "The dataset contains 1,565 research papers in PDF format. Any duplicate papers were removed automatically, though some could exist"

• Time frame: "The dataset was curated the duration of the research paper, primarily in February of 2024"

• Source distribution: "We wrote scripts to automatically query the APIs of Arxiv and Semantic Scholar"

Quality Control

• Human validation: "After collecting data from different sources, we removed duplicate papers and did a manual and semi-automated review of papers to ensure they were all relevant"

• LLM-assisted review: "We develop a prompt using gpt-4-1106-preview to classify the remaining articles. We validate the prompt against 100 ground-truth annotations, achieving 89% precision and 75% recall (for an F1 of 81%)"

Search Keywords (Selected Examples)

• Core terms: "jailbreak prompt", "prompt engineering", "few-shot learning", "in-context learning"
• Technique-specific: "chain-of-thought", "zero-shot prompting", "prompt optimization"
• Domain-specific: "llm prompting", "transformer model prompts", "multimodal prompting"

Critical Insights & Limitations

Nature of Prompting

• Black art acknowledgment: "This can be interpreted both optimistically and pessimistically. Optimistically, it demonstrates how improvements can arise through exploration and fortuitous discovery. On the pessimistic side, the value of duplicating the email in the prompt highlights the extent to which prompting remains a difficult to explain black art"

• Emergent vs discovered: "Many of the techniques described here have been called 'emergent', but it is perhaps more appropriate to say that they were discovered—the result of thorough experimentation, analogies from human reasoning, or pure serendipity"

Validation Challenges

• Lack of standardization: "The field is new, and evaluation is variable and unstandardized—even the most meticulous experimentation may suffer from unanticipated shortcomings, and model outputs themselves are sensitive to meaning-preserving changes in inputs"

• Transfer uncertainty: "As a result, we encourage the reader to avoid taking any claims at face value and to recognize that techniques may not transfer to other models, problems, or datasets"

Scope Limitations

• Focus restrictions: "To keep the work approachable to less technical readers and maintain a manageable scope... we only study task-agnostic techniques"

• Exclusions: "These decisions keep the work approachable to less technical readers and maintain a manageable scope"

Practical Implementation Notes

Prompt Template Best Practices

• Variable replacement: "A prompt template is a function that contains one or more variables which will be replaced by some media (usually text) to create a prompt"

• Context preservation: "It is often necessary to include additional information in the prompt... Additional Information is sometimes called 'context', though we discourage the use of this term as it is overloaded with other meanings in the prompting space"

Answer Extraction Strategies

• Verbalizer design: "For example, if we wish for a model to predict whether a Tweet is positive or negative, we could prompt it to output either '+' or '-' and a verbalizer would map these token sequences to the appropriate labels"

• Regex patterns: "Regexes are often used to extract answers. They are usually used to search for the first instance of a label. However, depending on the output format and whether CoTs are generated, it may be better to search for the last instance"

• Cascading approaches: "Sometimes outputs are so complicated that regexes won't work consistently. In this case, it can be useful to have a separate LLM evaluate the output and extract an answer"

Model Selection Considerations

• Guardrails interference: "A take-away from this initial phase is that the 'guard rails' associated with some large language models may interfere with the ability to make progress on a prompting task, and this could influence the choice of model for reasons other than the LLM's potential quality"

• Temperature settings: "For the two Self-Consistency results, we set temperature to 0.5, following Wang et al. (2022)'s guidelines. For all other prompts, a temperature of 0 was used"

Terminology Disambiguation

Conflicting Usages

• In-Context Learning ambiguity: "Note that the word 'learn' is misleading. ICL can simply be task specification–the skills are not necessarily new, and can have already been included in the training data"

• Brown et al. definitions: "Brown et al. (2020) seemingly offer two different definitions for ICL... However, they explicitly state that ICL does not necessarily involve learning new tasks"

• Prompt vs Prompt Template: "Brown et al. (2020) consider the word 'llama' to be the prompt, while 'Translate English to French:' is the 'task description'. More recent papers, including this one, refer to the entire string passed to the LLM as the prompt"

Hard vs Soft Prompts

• Hard (discrete): "These prompts only contain tokens that directly correspond to words in the LLM vocabulary"

• Soft (continuous): "These prompts contain tokens that may not correspond to any word in the vocabulary... Soft prompts can be used when fine-tuning is desired, but modifying the weights of the full model is prohibitively expensive"

Prefix vs Cloze

• Prefix prompts: "In Prefix prompts, the token to be predicted is at the end of the prompt. This is usually the case with modern GPT-style models"

• Cloze prompts: "In Cloze prompts, the token(s) to be predicted are presented as 'slots to fill', usually somewhere in the middle of the prompt. This is usually the case for earlier transformer models such as BERT"

Advanced Technique Details

AutoDiCoT (Novel Contribution)

• Algorithm description: "We call the algorithm in Figure 6.12 Automatic Directed CoT (AutoDiCoT), since it automatically directs the CoT process to reason in a particular way"

• Process: "For each pair (qi, ai) in training data: Label qi as entrapment or not using the model. If correct, prompt with 'Why?' to generate reasoning. If incorrect, prompt 'It is actually [is/is not] entrapment, please explain why.'"

• Generalizability: "This technique can be generalized to any labeling task. It combines the automatic generation of CoTs with showing the LLM examples of bad reasoning, as in the case of Contrastive CoT"

Design Decision Framework

• Six critical factors: "We highlight six separate design decisions, including the selection and order of exemplars that critically influence the output quality"

• Tradeoffs: "Although effective, employing KNN during prompt generation may be time and resource intensive"

Iterative Retrieval

• FLARE approach: "Forward-Looking Active REtrieval augmented generation (FLARE) and Imitate, Retrieve, Paraphrase (IRP) perform retrieval multiple times during long-form generation"

• Three-step process: "1) generating a temporary sentence to serve as a content plan; 2) retrieving external knowledge using the temporary sentence as a query; 3) injecting the retrieved knowledge into the temporary sentence"

• Query quality: "These temporary sentences have been shown to be better search queries compared to the document titles provided in long-form generation tasks"

Meta-Analysis Statistics

Citation Patterns

• Most cited techniques: "The prevalence of citations for Few-Shot and Chain-of-Thought prompting is unsurprising and helps to establish a baseline for understanding the prevalence of other techniques"

• Model usage: Citation analysis shows GPT family dominates research, followed by PaLM and open-source alternatives

• Dataset popularity: MMLU, GSM8K, and arithmetic reasoning benchmarks most frequently used

Research Trends

• Paper growth: 1,565 relevant papers identified from broader corpus of 4,247 unique records

• Quality metrics: Inter-annotator agreement of 92% (Krippendorff's α = Cohen's κ = 81%) for relevance labeling

• LLM assistance: "We validate the prompt against 100 ground-truth annotations, achieving 89% precision and 75% recall (for an F1 of 81%)" for automated paper screening

Formal Definitions

Mathematical Formulation

• Basic prompt conditioning: "p(A|T,Q) = ∏(i=1 to |A|) p_LM(ai|T,Q,a1:i-1)" where T is prompt template, Q is question, A is answer

• Few-shot extension: "p(A|T(X,x)) = ∏(i=1 to |A|) p_LM(ai|T(X,x),a1:i-1)" where X is set of training exemplars

• Optimization objective: "T* = argmax_T E_{xi,yi~D}[S(p_LM(A|T(xi)),yi)]" maximizing scoring function S over dataset D

• Answer engineering: "A ~ p_LM(A|T(xi),yi); T* = argmax_{T,E} E_{xi,yi~D}[S(E(A),yi)]" where E is extraction function

Storage & Implementation Constraints

Browser Environment

• Critical restriction: "NEVER use localStorage, sessionStorage, or ANY browser storage APIs in artifacts. These APIs are NOT supported and will cause artifacts to fail in the Claude.ai environment"

• Alternatives: "Instead, you MUST: Use React state (useState, useReducer) for React components; Use JavaScript variables or objects for HTML artifacts; Store all data in memory during the session"

Library Availability (React Artifacts)

• Available libraries include: lucide-react, recharts, MathJS, lodash, d3, Plotly, Three.js (r128), Papaparse, SheetJS, shadcn/ui, Chart.js, Tone, mammoth, tensorflow
• Important limitation: "NO OTHER LIBRARIES ARE INSTALLED OR ABLE TO BE IMPORTED"
• Three.js caveat: "IMPORTANT: Do NOT use THREE.CapsuleGeometry as it was introduced in r142. Use alternatives like CylinderGeometry, SphereGeometry, or create custom geometries instead"

Contributions & Authorship

Team Structure

• Lead authors: Sander Schulhoff (lead), Michael Ilie (co-lead)
• Principal investigator: Philip Resnik
• Total contributors: 58 authors from 13 institutions

Major Section Leads

• Benchmarking: Konstantine Kahadze
• Agents: Ashay Srivastava
• Alignment: Nishant Balepur
• Security: Sevien Schulhoff
• Multilingual: Dayeon Ki
• Evaluation: Sweta Agrawal

Domain Expertise

• SCS labeling: Megan L. Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker provided clinical expertise
• Multilingual guidance: Marine Carpuat framed and reviewed multilingual section

Additional Resources

Maintained Resources

• Live terminology: "We maintain an up-to-date list of terms and techniques at LearnPrompting.org"
• Dataset access: Available on HuggingFace with full datasheet
• Code repository: GitHub with systematic review pipeline

Future Updates

• Iterative taxonomy: "We expect this to be the first iteration of terminologies that will develop over time"
• Community contribution: "If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes, anyone is free to use/modify the data"

Citation Information

• Preferred citation: Schulhoff et al. (2024), "The Prompt Report: A Systematic Survey of Prompting Techniques"
• Contact: sanderschulhoff@gmail.com for dataset inquiries
• Funding acknowledgment: "$10,000 in API credits given by OpenAI"

RAG is Dead, Context Engineering is King — with Jeff Huber of Chroma

Core Thesis

• Context Engineering over RAG: "RAG" as a term is fundamentally flawed and confusing

  "RAG. We never use the term rag. I hate the term rag... retrieval, augmented generation. Are three concepts put together into one thing? Like, that's just really confusing."

• Context Engineering Definition: The job of determining optimal context window contents

  "Context engineering is the job of figuring out what should be in the context window any given LLM generation step. And there's both an inner loop, which is setting up the, you know, what should be in the context window this time. And there's the outer loop, which is how do you get better over time at filling the context window with only the relevant information."

Context Rot Research

• Models degrade with longer contexts: Performance is not invariant to token count

  "The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less and then also can reason sort of less effectively."

• Needle in Haystack is misleading: Lab benchmarks don't reflect real-world usage

  "There was this bit of, like, this sort of implication where, like, oh, look, our model is perfect on this task, needle in a haystack. Therefore, the context window you can use for whatever you want. There was an implication there. And, well, I hope that that is true someday. That is not the case."

• Claude Sonnet 4.5 performs best: Based on area under curve for context utilization

  "I don't have much commentary. That is what we found for this particular task... I think it shows here if this is true, that's a big explanation for why" developers love Claude

Retrieval System Architecture

First-Stage Retrieval (Hybrid Approach)

• Multiple signals for initial culling: Dense vectors, lexical search, metadata filtering

  "One pattern is to use what a lot of people call first stage retrieval to do a big cull down... using signals like vector search, like full text search, like metadata filtering, metadata search, and others to go from, let's say 10,000 down to 300."

• LLMs can handle more than 10 results: Unlike traditional search for humans

  "You don't have to give an LLM 10 blue links. You can brute force a lot more."

Re-ranking with LLMs

• LLM re-ranking is cost-effective and emerging: From 300 candidates down to 30

  "Using an LLM as a re-ranker and brute forcing from 300 down to 30, I've seen now emerging a lot, like a lot of people are doing this and it actually is like way more cost effective than I think a lot of people realize I've heard of people that are running models themselves that are getting like a penny per million input tokens"

• Purpose-built re-rankers will decline: Like specialized hardware, only needed at extreme scale

  "I actually think that like probably purpose built re-rankers will go away. And the same way that like purpose built... if you're at extreme scale, extreme cost, yes, you'll care to optimize that... the same way that if you're running with hardware... you're just going to use a CPU or GPU. Unless you absolutely have to."

Context Assembly Best Practices

• Structured ingestion matters: Extract metadata and signals at write time

  "As much structured information as you can put into your write or your ingestion pipeline, you should. So all of the metadata you can extract, do it at ingestion. All of the chunk rewriting you can do, do it at ingestion."

• Chunk rewriting for code: Generate natural language descriptions

  "Instead of just embedding the code, you first have an LLM generate like a natural language description of like what this code is doing. And either you embed like just the natural language description or you embed that and the code"

Code Retrieval Strategies

• Regex remains dominant: 85-90% of queries satisfied, but embeddings add value

  "My guess is that like for code today, it's something like 90% of queries or 85% of queries can be satisfactorily run with Regex... But you maybe can get like 15% or 10% or 5% improvement by also using embeddings."

• Chroma supports native regex search: With indexing for scale

  "We've actually worked on now inside of Chroma, both single load and distributed, we support regex search natively. So you can do regex search inside of Chroma because we've seen that as like a very powerful tool for code search."

• Fork-based indexing for versioning: Fast branch/commit-specific indexes

  "Another feature we added to Chroma is the ability to do forking. So you can take an existing index and you can create a copy of that index in under a hundred milliseconds for pennies... you now can like have an index for like different each commit."

Generative Benchmarking

• Small golden datasets are highly valuable: Few hundred examples sufficient

  "The returns to a very high-quality small label data set are so high. Everybody thinks you have to have, like, a million examples or whatever. No. Actually, just, like, a couple hundred even, like, high-quality examples is extremely beneficial."

• Generate synthetic QA pairs: When you have chunks but need queries

  "We did a whole technical report around how do you teach an LLM to write good queries from chunks? Because, again, you want, like, chunk query pairs. And so if you have the chunks, you need the queries."

• Data labeling parties work: Simple, practical approach

  "Thursday night, we're all going to be in the conference room. We're ordering pizza. And we're just going to have a data labeling party for a few hours. That's all it takes to bootstrap this."

Memory and Context Engineering

• Memory is context engineering's benefit: Same problem, different framing

  "Memory again is like the memory is the term that like everybody can understand... but what is memory under the hood? It's still just context engineering... the domain of how do you put the right information into the context window?"

• Compaction enables offline improvement: Re-indexing and refinement

  "Offline processing is helpful, and I think that is also helpful in this case... You're taking data. You're like, oh, maybe those two data points should be merged. Maybe they should be split. Maybe they should be, like, rewritten."

Future of Retrieval Systems

• Stay in latent space: Avoid natural language round-trip

  "Why are we going back to natural language? Why aren't we just like passing the embeddings like directly to the models who are just going to functionally like re put it into latent space."

• Continuous retrieval during generation: Not just one-shot before generation

  "For the longest time we've done one retrieval per generation... why are we not continually retrieving as we need to"

• Current approaches are crude: Will seem primitive in 5-10 years

  "I think when we look back in things, this was like, like hilariously crude, the way we do things today."

Chroma Product Philosophy

• Developer experience is paramount: Zero-config, serverless approach

  "In the same way that you could run pip install ChromaDB and be up and running in five seconds... That same story had to be true for the cloud... It needed to be like zero config, zero knobs to tune. It should just be always fast, always very cost-effective and always fresh without you having to do or think about anything."

• Usage-based billing: True serverless pricing

  "We only charge you for the minimal slice of compute that you use and like nothing more, which not all serverless databases can claim"

• Slow, intentional hiring: Culture and craft over speed

  "The slope of our future growth is entirely dependent on the people that are here in this office... we've just really decided to hire very slowly and be really picky."

Key Technical Reports

1. Context Rot: LLM performance degradation with context length
2. Generative Benchmarking: Synthetic QA pair generation for evaluation

Referenced Papers/Technologies

• REALM: Retrieval-Augmented Language Model
• RETRO: Retrieval-Enhanced Transformer
• Voyager paper: Used Chroma in implementation
• Lance Martin's Context Engineering: Key thought leadership
• TLA Plus, Raft, Paxos: Distributed systems foundations

Company Details

• Downloads: 5M+ monthly, 70M+ all-time on PyPI
• GitHub: 21,000+ stars
• Architecture: Rust-based, fully multi-tenant, separation of storage/compute
• Open Source: Apache 2 license for core and distributed versions
• Cloud: Serverless, usage-based, $5 free credits (~100K docs + queries)

Advanced Context Engineering for Agents - Summary

Overview

• Source: https://www.youtube.com/watch?v=IS_y40zY-hc
• Type: Technical Conference Talk
• Length: ~14 minutes (YC Root Access)
• Speaker: Dexter Horthy, Founder of Human Layer (YC Fall 24)
• Key Focus: Advanced context engineering techniques for scaling coding agents in production environments

Executive Summary

Dexter Horthy presents a systematic approach to context engineering that transforms AI coding from prototyping to production-ready development. He demonstrates how spec-first development, intentional context management, and structured workflows enable teams to ship complex code in large repositories while maintaining quality and team alignment.

Key Insights

• Context as Core Constraint: "LLMs are pure functions. The only thing that improves the quality of your outputs is the quality of what you put in, which is your context window." - Context management is the fundamental lever for agent performance
• Spec-First Development: "In the future where AI is writing more and more of our code, the specs, the description of what we want from our software is the important thing." - Specifications become the source code equivalent in AI-driven development
• Hierarchy of Impact: "A bad line of research, a misunderstanding of how the system works and how data flows and where things happen can be thousands of bad lines of code." - Early-stage errors compound exponentially through the development process

Key Elements (CRITICAL FOR LOOKUP)

Key Concepts

• Context Engineering: "Everything that makes agents good is context engineering" - [Core philosophy throughout talk]
• Intentional Compaction: "Be very intentional with what you commit to the file system and the agents memory" - [08:48 timestamp]
• Spec-First Development: "We were forced to adopt spec first development because it was the only way for everyone to stay on the same page" - [03:12 timestamp]
• 40% Context Rule: "Our goal all the time is to keep context utilization under 40%" - [11:00 timestamp]
• Research-Plan-Implement Workflow: "We have three phases research, plan and implement" - [11:00 timestamp]

Key Personalities

• Dexter Horthy: "My name is Dex. I'm the founder of a company called Human Layer" - [Speaker, YC Fall 24]
• Sean Grove: "Sean Grove, the new code. He talked about how we're all vibe coding wrong" - [Referenced expert on coding practices]
• Jeff Huntley: "Jeff Huntley works on source AMP... he wrote this thing called Ralph Wigum as a software engineer" - [Context optimization expert]
• Vibbov: "I do a podcast with another YC founder named Vibbov. He built Bam" - [Collaboration partner, BAML creator]

Key Tools/Technologies

• Human Layer: "I'm the founder of a company called Human Layer" - [Dexter's company focused on context engineering]
• BAML: "He built Bam... has anyone here you used BAML before?" - [Programming language/tool for AI workflows]
• Sub Agents: "A lot of people saw cloud code sub aents and they jumped in... but they're really about context control" - [Context management technique]
• MCP Tools: "If you have MCP tools that return big blobs of JSON, that's going to flood your context window" - [Tool integration consideration]

Key References

• 12 Factor Agents: "We wrote a weird little manifesto called 12actor agents um principles of reliable LLM applications" - [April 22nd foundational work]
• Stanford Study: "The Stanford study... they ingested data from 100,000 developers... AI engineering and software leads to a lot of rework" - [Research on AI coding effectiveness]
• Ralph Wigum Article: "He wrote this thing called Ralph Wigum as a software engineer" - [Context optimization methodology]
• Open Source Prompts: "This is our research prompt. It's really long. It's open source. You can go find it" - [Available implementation resources]

Detailed Analysis

The Problem with Current AI Coding

• Naive Approach Fails: "The most naive way to use a coding agent, which is to shout back and forth with it until you run out of context or you give up or you cry" - [04:48]
• Complex Systems Challenge: "Doesn't work in big repos, doesn't work for complex systems" - [02:44]
• Rework Problem: "AI engineering and software leads to a lot of rework. So even if you get benefits, you're actually throwing half of it away" - [01:45]

Context Engineering Solutions

• Intentional Compaction Strategy: "Even if we're on the right track, if we're starting to run out of context, be very intentional with what you commit to the file system and the agents memory" - [05:45]
• Sub-Agent Context Control: "The parent agent can get right to work without having to have the context burden of all of that reading and searching" - [07:27]
• Frequent Compaction Workflow: "Building your entire development workflow around context management" - [08:48]

Three-Phase Implementation

• Research Phase: "Understand how the system works and all the files that matter and perhaps like where a problem is located" - [11:00]
• Planning Phase: "Tell me every single change you're going to make. not line by line, but like include the files and the snippets" - [11:12]
• Implementation Phase: "If the plan is good, I'm never shouting at cloud cloud anymore. And if I'm shouting at cloud, it's because the plan was bad" - [11:59]

Actionable Takeaways

1. Implement Spec-First Development: Start with detailed specifications before any code generation
2. Maintain 40% Context Utilization: Keep context windows under 40% capacity for optimal performance
3. Use Three-Phase Workflow: Structure all development as Research → Plan → Implement
4. Review Plans, Not Code: Focus human review on specifications and plans rather than generated code
5. Implement Intentional Compaction: Regularly compress context with structured progress files

Technical Details

• Tools/Technologies: Human Layer, BAML, Sub-agents, MCP tools, Context compaction systems
• Requirements: ~170,000 token context windows, structured prompt engineering, team workflow transformation
• Implementation Notes: Open-source prompts available, requires significant team process changes

Case Study Results

• BAML Rust Codebase: "We decided to see if we could oneshot a fix to a 300,000 line RS codebase... The PR was so good the CTO did not know I was doing it as a bit and he had merged it" - [11:12]
• Boundary CEO Session: "For 7 hours we sat down and we shipped 35,000 lines of code... he estimated that was 1 to two weeks of work roughly" - [12:44]
• Team Productivity: "Our intern Sam... shipped two PRs on his first day. on his eighth day, he shipped like 10 in a day" - [13:30]

Open-Source Prompts Discovery

FOUND! The research and planning prompts Dexter mentioned are available in Human Layer's GitHub repository:

Research Prompt

• Location: https://github.com/humanlayer/humanlayer/blob/main/.claude/commands/research_codebase.md
• Purpose: Comprehensive codebase research using parallel sub-agents
• Key Features:
• Spawns specialized agents (codebase-locator, codebase-analyzer, thoughts-locator, etc.)
• Structured research document generation with YAML frontmatter
• File path and line number references for developer navigation
• Integration with thoughts directory for historical context

Planning Prompt

• Location: https://github.com/humanlayer/humanlayer/blob/main/.claude/commands/create_plan.md
• Purpose: Interactive implementation plan creation through iterative process
• Key Features:
• 5-step process: Context Gathering → Research & Discovery → Plan Structure → Detailed Writing → Sync & Review
• Automated vs Manual success criteria separation
• Phase-based implementation with specific file changes and verification steps
• Integration with specialized research agents

Implementation Methodology

These prompts demonstrate the practical application of Dexter's three-phase workflow:

1. Research Phase: Uses research_codebase.md to understand system architecture
2. Planning Phase: Uses create_plan.md to create detailed implementation specifications
3. Implementation Phase: Structured execution with clear success criteria

References & Follow-up

• Related Topics: Prompt engineering, AI workflow optimization, team transformation strategies
• Source Links:
• YouTube video: https://www.youtube.com/watch?v=IS_y40zY-hc
• Research Prompt: https://github.com/humanlayer/humanlayer/blob/main/.claude/commands/research_codebase.md
• Planning Prompt: https://github.com/humanlayer/humanlayer/blob/main/.claude/commands/create_plan.md
• Human Layer Repository: https://github.com/humanlayer/humanlayer
• Quote Locations: Timestamps provided throughout for key concepts and claims

• Context Engineering is Paramount: The central thesis is that the quality of an AI agent's output is entirely dependent on the quality of the input context. Improving agents is a matter of improving the context you provide them. > "The only thing that improves the quality of your outputs is the quality of what you put in, which is your context window."

• Critique of Naive Agent Usage: The speaker criticizes the common practice of iteratively prompting an agent without a structured plan. He likens this to writing code, compiling it, and then throwing away the source code, as the valuable "spec" (the prompts and conversation) is lost. > "the idea of sitting and talking to an agent for two hours and figuring out and exactly specifying what you want to do and then throwing away all the prompts and committing the code is basically equivalent to... you checked in the compiled asset and you threw away the code."

• The "Spec-First" Workflow: To manage the complexity of large, AI-generated pull requests, the speaker's team adopted a "spec-first" development process. This shifts the focus from reviewing code to reviewing detailed plans and research documents. > "we were forced to adopt spec first development because it was the only way for everyone to stay on the same page."

• Three-Phase Context Management: The core of their process involves three distinct phases, each designed to create high-quality context for the next:

  1. Research: The agent first explores the codebase to understand the system, identifying relevant files and logic. The output is a research document.
  2. Plan: Based on the research, a detailed implementation plan is created, specifying all intended changes, files to be modified, and testing strategies.
  3. Implement: The agent executes the plan, with the context window kept clean and focused (under 40% utilization) by progressively marking parts of the plan as complete.

     "we have three phases research, plan and implement."

• The Hierarchy of Leverage: The talk emphasizes that errors in the early stages have a cascading effect. A mistake in the research phase can lead to thousands of lines of incorrect code, making the research and planning documents the most critical artifacts to review. > "a bad line of code is a bad line of code. And a bad part of a plan can be hundreds of bad lines of code. And a bad line of research... can be thousands of bad lines of code."

• Redefining Code Review: Code review's most important function is maintaining mental alignment within a team. Reviewing concise, well-structured plans is more effective for this than trying to parse thousands of lines of AI-generated code. > "code review is about a lot of things, but the most important part is mental alignment."

• Proven Results: This methodology has allowed the team to solve complex problems and ship massive amounts of code at an accelerated pace, including successfully fixing a bug in a 300,000-line Rust codebase in a single attempt. > "we did get it merged. The PR was so good the CTO did not know I was doing it as a bit and he had merged it by the time we were recording the episode."

• Future Outlook: The speaker predicts that the technology of coding agents will become a commodity. The true differentiator for teams will be their ability to adapt their workflows and communication to effectively harness these tools.

  "I kind of maybe think coding agents are going to get a little bit commoditized, but the team and the workflow transformation will be the hard part."

https://bytes.vadeai.com/how-one-clojure-function-destroyed-agent-framework-completely/

How Clojure's iteration Function Disrupts Agent Frameworks

Agent Framework Pitfalls

Traditional agent frameworks like CrewAI introduce significant complexity with configuration files, rigid agent definitions, and orchestration mechanics. They require developers to manage state, orchestrate tasks, handle errors, and manage resources, but still abstract away critical decisions, making debugging and customization difficult[1].

The Simplicity of iteration

Clojure 1.11 introduced the iteration function, which models any sequential, stateful process — including agentic workflows — far more simply than the framework approach. Its key parameters:

• step: Does the work (e.g., LLM call, tool execution)
• initk: Starting state (prompt, initial data)
• vf: Extracts the meaningful result from each step
• kf: Determines the next state for the following iteration
• somef: Decides if the workflow continues or stops

This aligns perfectly with agentic workflows: - step: agent action - initk: initial task/state - vf: extract agent output - kf: update agent context/state - somef: goal/termination checker[1]

Real World Example

A basic agent workflow with iteration in Clojure:

clojure (defn simple-agent-workflow [initial-prompt max-iterations] (let [llm-instance (create-llm-instance) step-fn (fn [{:keys [iteration prompt response]}] (when (< iteration max-iterations) (let [messages [(create-message :user prompt)] new-response (generate llm-instance messages) next-prompt (extract-next-task new-response)] {:iteration iteration :prompt prompt :response new-response :next-token {:iteration (inc iteration) :prompt next-prompt :response new-response}})))] (iteration step-fn :somef (fn [res] (some? res)) :vf identity :kf :next-token :initk {:iteration 0 :prompt initial-prompt :response {}})))

Production variants in Vade AI simply expand this pattern for live API streaming, logging, and complex branching, without introducing unnecessary abstraction or opaque state[1].

Benefits Over Frameworks

• Complete control: Every workflow step and state transition is transparent and customizable.
• Easy debugging: Print or inspect state at any moment; no special debugging tools needed.
• Flexible termination: Workflow can halt based on any custom logic, not just predefined callbacks.
• Resource efficiency: No framework overhead, predictable and low memory footprint.
• Streaming and real-time: Can process operations incrementally as LLM responses stream in, with immediate visibility for the user[1].
• Composability: Integrates natively with the rest of Clojure — no framework lock-in.

When to Use This Approach

The iteration pattern is ideal when you need:

• Custom agent behaviors
• Transparent workflows
• Performance optimization
• Complex branching or termination logic
• Deep integration with Clojure systems

Especially powerful for research and analysis, planning systems, validation pipelines, and unique business logic that standard frameworks struggle to express[1].

Key Takeaways

Frameworks often create more complexity than they solve. By embracing Clojure’s iteration, you implement agentic workflows with less code, greater clarity, and full control. This enables adaptive, resource-aware, and highly debuggable systems — proven at scale inside Vade AI[1].

Citations: [1] How One Clojure Function Destroyed Agent Framework Completely https://bytes.vadeai.com/how-one-clojure-function-destroyed-agent-framework-completely/

• Purpose of the Smalltalk project

  "The purpose of the Smalltalk project is to provide computer support for the creative spirit in everyone."

• Emphasis on enabling creativity via computing hardware and software.

• Focus of research areas

  "We have chosen to concentrate on two principle areas of research: a language of description (programming language) ... and a language of interaction (user interface)..."

• Programming language as an interface between mental models and hardware.

• User interface as a communication bridge between human and computer systems.

• Development process

  "Our work has followed a two- to four-year cycle that can be seen to parallel the scientific method."

• Iterative cycles aligned with hypothesis-experiment-validation.

• Two levels of communication

  "Explicit communication includes the information that is transmitted in a given message. Implicit communication includes the relevant assumptions common to the two beings."

• Distinction between explicit message content and shared implicit context.

• System complexity and dependencies

  "If there are N components in a system, then there are roughly N-squared potential dependencies between them."

• Larger systems exponentially increase chances for unwanted interactions.

• Design goal to minimize interdependence

  "...a computer language should be designed to minimize the possibilities of such interdependence."

• Isolation of components reduces complexity-related risks.

• Message-sending metaphor for modularity

  "The message-sending metaphor provides modularity by decoupling the intent of a message ... from the method used by the recipient to carry out the intent."

• Separates communication intent from implementation detail.

• Protection of structural information

  "All access to the internal state of an object is through this same message interface."

• Encapsulation enforced through message-based access.

• Reducing system complexity via grouping

  "The complexity of a system can often be reduced by grouping similar components."

• Logical grouping organizes structure and reduces mental overhead.

• Smalltalk’s class mechanism

  "A class describes other objects — their internal state, the message protocol they recognize, and the internal methods for responding to those messages."

• Classes define state, protocol, and behavior for their instances.

• Instances and meta-classes

  "Even classes themselves fit into this framework; they are just instances of class Class..."

• Meta-level reflection: classes are themselves objects.

• Nature of a user interface

  "A user interface is simply a language in which most of the communication is visual."

• UI framed as a predominantly visual language.

• Role of esthetics in UI

  "Because visual presentation overlaps heavily with established human culture, esthetics plays a very important role..."

• Design aesthetics influence user understanding and comfort.

• Flexibility in UI design

  "Since all capability of a computer system is ultimately delivered through the user interface, flexibility is also essential here."

• UI adaptability is critical for capability delivery.

• Object-oriented principle enabling UI flexibility

  "An enabling condition for adequate flexibility of a user interface can be stated as an object-oriented principle."

• OOP structures support adaptable interface design.

Summary of the Discussion on SwiftUI: Understanding Identity, Lifetime, and Dependencies

1. Introduction to SwiftUI and Its Declarative Nature

2. SwiftUI operates as a declarative UI framework where you describe UI states, and the framework manages their actualization. "That means that you describe what you want for your app at a high level, and SwiftUI decides exactly how to make it happen."

3. Understanding Identity in SwiftUI

4. SwiftUI views have identity to distinguish elements as the same or different across updates, critical for UI transitions and state management. "Identity is how SwiftUI recognizes elements as the same or distinct across multiple updates of your app."

5. Concept of View Identity Using Practical Examples

6. Demonstrated using the "Good Dog, Bad Dog" app example, explaining how identity influences view transitions and behavior. "That distinction actually matters a great deal because it changes how our interface transitions from one state to another."

7. Explicit vs. Structural Identity

8. Discussed two types of identity:

   • Explicit identity is assigned using identifiers like tags. "Explicit identity is powerful and flexible, but does require that someone, somewhere keeps track of all those names."
   • Structural identity is derived from the view's type and position in the hierarchy. "SwiftUI uses the structure of your view hierarchy to generate implicit identities for your views so you don't have to."

9. Role of Lifetime in SwiftUI

10. Explained how SwiftUI manages the life cycle of views and data by associating views' identity over time. "Lifetime is how SwiftUI tracks the existence of views and data over time."

11. Impact of Dependencies on UI Updates

12. Dependencies are inputs like state variables or environmental settings that trigger UI updates when they change. "Dependencies are how SwiftUI understands when your interface needs to be updated and why."

13. How SwiftUI Manages State and Identity

14. Discussed how State and StateObject help preserve state across the lifetime of views tied to their identity. "State and StateObject are the persistent storage associated with your view's identity."

15. Advanced Use of Identity with SwiftUI's ForEach

16. ForEach leverages identifiers for efficient updates and animations, showing how identity can impact performance and correctness. "Choosing a good identifier is your opportunity to control the lifetime of your views and data."

17. Best Practices for Using Identity

18. Emphasized the importance of stable and unique identifiers to improve performance and prevent state loss. "An identifier that isn't stable can result in a shorter view lifetime."

19. Troubleshooting and Optimization Techniques

    • Discussed common pitfalls with AnyView and alternatives using view builders to optimize SwiftUI’s understanding and performance. "Having too many AnyViews will often make code harder to read and understand."

Key Takeaways

• Identity, lifetime, and dependencies are core concepts that determine how SwiftUI manages and updates the UI.
• Effective management of these properties can significantly enhance the performance and predictability of SwiftUI applications.
• Developers are encouraged to use stable and unique identifiers and understand the implications of explicit and structural identities on their code.

This summary focuses on the critical aspects discussed in the tech talk, ensuring a comprehensive understanding of the primary themes and practical implications for SwiftUI developers.

• Introduction

• Eric Normand introduces himself and the purpose of the talk: "The title of this talk is building composable abstractions...to develop a process to do that and also I'd like to start a discussion about how we can do that better."

• Importance of Abstractions

• Abstractions are critical for creating complex applications from small, simple problems. "A lot of people are able to solve small problems like Fibonacci...when they finally want to create an app...they don't know how to take the tools that they've learned and turn them into software."

• Map of the Talk

• The talk covers the importance of abstractions, the process of developing them, an example, and concluding thoughts. "Here's sort of the map of the talk: why focus on abstractions, the process, an example abstraction, and concluding thoughts."

• Why Focus on Abstractions?

• Refactoring introduces the distinction between the behavior of the code and its implementation. "In the general industry we now have this idea that there's a difference between the behavior of the code and the actual implementation."

• Example of Newtonian mechanics replacing Aristotelian physics illustrates that some systems can't be refactored but need to be redesigned from scratch. "You can't refactor Aristotle into Newton."

• Objectives of the Abstraction Process

• The process should produce good, Newtonian-style abstractions, be iterative, accessible to all, and foster collaboration. "It has to consistently produce good abstractions...an iterative process...anyone can do it...fosters collaboration."

• Example of Vector Graphics System

• Normand uses a simple vector graphics system as an example to demonstrate the process of building abstractions. "This is the example we're going to develop: a vector graphics system."

• Step 1: Physical Metaphor

• Choose a metaphor to capture important information. "The idea behind this is to choose a metaphor that will capture the important information in your program."

• Shapes and construction paper is chosen as the metaphor. "Shapes and construction paper...I cut out shapes like rectangles and ellipses...and then I can move them around."

• Step 2: Meaning Construction

• Convert physical intuition into precise mathematical language, focusing on the interface. "We're going to be focusing on the interface right now...precise mathematical language."

• Definitions in Clojure for different components like color, shape, and transformations. "We're defining two types here: cutout and shape...defining a function that takes a cutout and returns a shape."

• Importance of preserving shape and color, overlay order, and rotation and translation independence. "Preservation of shape...preservation of color...overlay order...rotation and translation independence."

• Step 3: Implementation

• Implement the system based on the constructed meaning, ensuring it can be refactored to different requirements like SVG output. "Implementation...we already know what to do...refactor from quill to SVG."

• Summary of Process

• Use a physical metaphor, define the parts and their relationships in mathematical language, and refactor for implementation details. "Use a physical metaphor...define the parts and their relationships...refactor to get all the meta properties."

• Corollaries for the Process

• Know your domain, constructs, and refactoring techniques. "Know your domain...know your constructs...know your refactoring."

• Conclusion

• Encourages further learning and provides resources. "Please go to my site...download the slides...sign up for my newsletter."

• ClojureScript has excelled in standard UI patterns but now aims to harness modern browser APIs for high-performance use cases.

"The majority of Clojurescript application development and community discussions seems to be focused on improving standard UI implementation patterns and the general workflow of how we can build web applications better, easier and faster." (Medium)

• The workshop’s goal was to probe ClojureScript’s internals, identify bottlenecks, and introduce technologies like WebGL, WebRTC, WebWorkers, and Emscripten.

"So for this workshop I chose to look more below the surface of Clojurescript, analyze problem areas, examine possible optimization strategies and above all introduce people to a number of modern web technologies (WebGL, WebRTC, WebWorkers, Emscripten), techniques & tools offering possible routes to use the language in a sound and elegant way to work with these features." (Medium)

• A six-step implementation of Conway’s Game of Life, from naive to optimized, achieved a speedup from 10,840 ms to 16.5 ms per frame (\~650× faster) on a 1024×1024 grid.

"Six implementations of Conway’s Game of Life — from naive (but idiomatic & slow) to optimized Clojurescript using typed arrays and direct pixel manipulations (10,840 ms / frame vs 16.5 ms / frame = \~650× faster for a 1024×1024 grid)." (Medium)

• A compile-time macro version of get-in eliminated temporary vector allocations and reduce calls, boosting lookup speed from 205.18 ns to 43.61 ns (\~5× faster).

"Benchmarking this example with criterium under Clojure (which has somewhat different/faster protocol dispatch than in Clojurescript), the macro version results in 43.61ns vs 205.18ns for the default get-in (\~5× faster)." (Medium)

• Switching from nested vectors to a flat 1D vector enabled nth-based indexing (\~6× speed-up), before Typed Arrays and loop-based pixel updates removed millions of function calls for the full \~650× gain.

"The more obvious improvement to speed up the simulation was using a flat 1D vector to encode the grid and calculate cell indices for the 2D coordinates ... gain a \~6× speed up ... Since all our data ... are stored in typed arrays ... and altogether gained a \~650× speedup compared to the original." (Medium)

• Adopting transduce for neighbor counting proved \~15–20% slower than map & reduce, highlighting that idiomatic functions can sometimes underperform.

"One of the intermediate steps ... was using transduce instead of map & reduce to compute the number of alive neighbor cells, however this ended up actually being \~15–20% slower in this case." (Medium)

• Effective WebGL programming demands deep knowledge of geometry, linear algebra, the OpenGL state machine, GPU pipelines, and GLSL, making it daunting for newcomers.

"To anyone interested in directly utilizing the GPU in the browser, WebGL is a huge & fascinating topic, but it can also be very daunting for newcomers to graphics programming, since efficient use of it requires a multitude of prerequisite knowledge and terminology about 2D/3D geometry, linear algebra, spatial thinking in multiple spaces (coordinate systems), low-level data organization, the OpenGL state machine ... GPU processing pipelines, knowledge of the GLSL shading language, color theory etc." (Medium)

• The thi.ng/geom library employs Clojure maps for semi-declarative OpenGL/WebGL buffer and shader specifications, while preserving explicit control over the GL state machine.

"The thi.ng/geom library takes a semi-declarative approach to working with OpenGL/WebGL in that it’s extensively using Clojure maps to define various geometry and shader specifications, which are then compiled into the required data buffers & GLSL programs ... but at no point is it hiding the underlying layer, giving advanced users full control over the GL state machine." (Medium)

• Shadergraph addresses GLSL code reuse by offering transitive dependency resolution, a library of common functions, compile-time minification, and metadata extraction for tooling.

"To address this in Clojurescript from early on, we can use the thi.ng/shadergraph library, which provides us with: a transitive dependency resolution mechanism for GLSL code ... a growing library of pure, commonly used GLSL functions (lighting, color conversion, matrix math, rotations, effects etc.) ... and a basic compile-time shader minifier ... Clojure meta data extraction of the defined GLSL functions ..." (Medium)

• A hands-on WebRTC demo showed how to stream a camera feed into Shadertoy-style WebGL shaders for real-time video FX processing.

"I prepared a small example combining a WebRTC camera stream with Shadertoy-like WebGL image processing using a bunch of effect options." (Medium)

• True parallelism in the browser comes from WebWorkers—unlike core.async’s simulated concurrency—and relies on isolated modules, message passing, and transferable ArrayBuffers for efficient data sharing.

"However, the currently only way to obtain real extra compute resources of a multi-core CPU in JavaScript is to use WebWorkers ... WebWorker code needs to be loaded from a separate source file and can only communicate with the main process via message passing. By default, the data passed ... is copied, but some types (e.g. ArrayBuffers) can also be transferred ..." (Medium)

• Emscripten’s LLVM-based compiler targets asm.js (and soon WebAssembly), enabling C/C++ modules to outperform idiomatic ClojureScript for math-heavy and mutable-data tasks.

"Emscripten ... a LLVM-based transpiler for C and C++ to asm.js ... the resulting asm.js code almost always performs noticeably faster than the Clojurescript version ... With WebAssembly on the horizon, it’s maybe a good time to invest some time into some “upskilling” ..." (Medium)

• The workshop’s capstone was a C-based 3D particle system demo, using Emscripten’s JavaScript ArrayBuffer heap and typed arrays to pack 36-byte particle structs tightly and avoid copying overhead.

"For the final exercise ... we implemented a simple 3D particle system in C, compiled it with Emscripten and learned how to integrate it into a Clojurescript WebGL demo ... The Emscripten runtime emulates the C heap as a single, large JS ArrayBuffer ... Each particle only takes up 36 bytes ... all particles in this array are tightly packed ..." (Medium)

"one of the best critiques of modern AI design comes from a 1992 talk by the researcher Mark Weiser where he ranted against “copilot” as a metaphor for AI."

• Weiser’s critique of the “copilot” metaphor for AI is foundational to this argument.

"He gave this example: how should a computer help you fly a plane and avoid collisions?... The agentic option is a 'copilot' — a virtual human who you talk with to get help flying the plane. If you’re about to run into another plane it might yell at you 'collision, go right and down!'"

• The “copilot” model is described as an interactive assistant giving explicit instructions or alerts.

"design the cockpit so that the human pilot is naturally aware of their surroundings. In his words: 'You’ll no more run into another airplane than you would try to walk through a wall.'"

• Weiser advocates for UIs that naturally augment human situational awareness, eliminating the need for explicit assistant intervention.

"Weiser’s goal was an 'invisible computer'—not an assistant that grabs your attention, but a computer that fades into the background and becomes 'an extension of [your] body.'"

• The ultimate aim: seamless, ambient support that integrates with human perception.

"the Head-Up Display (HUD), which overlays flight info like the horizon and altitude on a transparent display directly in the pilot’s field of view."

• The HUD is presented as a practical realization of Weiser’s concept: information is ambiently present, not actively disruptive.

"A HUD feels completely different from a copilot! You don’t talk to it. It’s literally part invisible—you just become naturally aware of more things, as if you had magic eyes."

• HUDs differ fundamentally from copilots by passively enhancing awareness instead of communicating via dialogue.

"spellcheck isn’t designed as a 'virtual collaborator' talking to you about your spelling. It just instantly adds red squigglies when you misspell something! You now have a new sense you didn’t have before. It’s a HUD."

• Spellcheck is given as a familiar analogy for AI-as-HUD: subtle, always-on augmentation of cognition.

"use AI to build a custom debugger UI which visualizes the behavior of my program!... With the debugger, I have a HUD! I have new senses, I can see how my program runs."

• Custom visual UIs for debugging exemplify HUD-like designs in AI-driven tools, enabling deeper understanding rather than focusing on transactional automation.

"Both the spellchecker and custom debuggers show that automation / 'virtual assistant' isn’t the only possible UI. We can instead use tech to build better HUDs that enhance our human senses."

• Non-agentic, perception-extending UI paradigms are positioned as powerful, sometimes preferable alternatives to assistant-like AI.

"I don’t believe HUDs are universally better than copilots!... anyone serious about designing for AI should consider non-copilot form factors that more directly extend the human mind."

• While not dismissing assistants, the author stresses the importance of exploring HUD-like, sense-extending UI for ambitious AI design.

"routine predictable work might make sense to delegate to a virtual copilot / assistant. But when you’re shooting for extraordinary outcomes, perhaps the best bet is to equip human experts with new superpowers."

• The conclusion is a nuanced tradeoff: assistants excel in predictable routines, but empowering human expertise requires HUD-style augmentation.

• Core Concept: The paper introduces Universalis, an AI-first, program-synthesis framework and language designed to be read by human knowledge workers but generated by AI models (LLMs). > "this paper outlines the high-level design of an AI-first, program-synthesis framework built around a new programming language, Universalis, designed for knowledge workers to read, optimized for our neural computer (Automind) to execute, and ready to be analyzed and manipulated by an accompanying set of tools."

• Design Philosophy: Universalis prioritizes readability for domain experts over writability for professional developers, making code intuitive and easier for AI to generate accurately. > "Unlike traditional programming languages, which prioritize syntax and structure optimized for writing by professional developers, Universalis is designed with the philosophy that code should be read by domain experts and written by machines."

• Structure and Syntax: The language syntax is analogous to "literate Excel spreadsheet formulas," embedding logical predicates inside [...] hedges surrounded by natural language explanations. > "Think of Universalis clauses as some kind of literate Excel spreadsheet formulas such as [@D is (@S-@B)] over named tables, or relations, enclosed in hedges surrounded by natural language explanation, where cells in the table correspond to variables such as @B, @S, and @D..."

• AI Safety through Contracts: It natively embeds pre- and post-conditions (contracts) into the language, providing a formal and extensible method to ensure the logical correctness and safety of AI-generated computations. > "By embedding pre- and post-conditions directly into the language semantics, Universalis provides a pragmatic and extensible method for implementing AI safety."

• Readable Conditionals: Conditional logic is structured as simple checklists, making decision-making processes clear and intuitive for human readers while still being executable by the AI interpreter, Automind. > "By structuring conditionals as checklists and explaining each branch in natural language, Universalis ensures that the logic is clear and intuitive for the human reader, while the Universalis interpreter Automind can still recognize them as control-flow decision points..."

• Loopless Bulk Processing: The framework handles operations on data collections via implicit broadcasting, similar to NumPy or modern Excel, removing the need for explicit loops that can be a distraction for non-programmers. > "In Universalis, this is handled by implicitly broadcasting operations on single elements to collections, similar to how NumPy or Pandas operate in Python or how dynamic array formulas and spilled array behavior in Excel allow for loopless programming."

• Accessible Data Queries: It features query comprehensions that use a structured, natural-language style for complex data operations like filtering and aggregation, making them more approachable than traditional query languages like SQL. > "By focusing on a structured natural-language approach for comprehending queries, Universalis ensures that even those with minimal experience in programming can perform advanced data manipulations."

• Simplified Data Extraction: The language includes powerful pattern-matching capabilities, allowing users to easily extract specific information from complex, nested data structures like JSON without writing complicated parsing code. > "This is where Universalis excels with its pattern-matching capability. Users can simply specify the patterns they want to match within the JSON structure..."

• Intentional Representation: Rather than generating final, concrete syntax, the Automind system produces an abstract, intentional representation of the user's intent, which can then be rendered into different formats. > "Since Universalis programs are trying to capture the high-level intent of the user as well, Automind does not generate the concrete syntax seen in the examples so far but instead creates an abstract, intentional representation of the Universalis code..."

• Minimalist Language Design: The language is intentionally kept minimal, focusing on core features like sequential composition, implicit looping, and nested dataframe queries to maintain readability and compatibility with formal verification tools. > "For Universalis, we intentionally keep the language minimal, focusing on sequential composition, implicit looping by lifting singleton operations over collections, and fully nested dataframe queries."

The Utopian Vision of Smalltalk

Smalltalk as a vision, not just a language:

“Smalltalk is a vision of the world as it should be, not necessarily as it is but the way it's supposed to be.”

A future imagined where Smalltalk has won:

“As we all know, Smalltalk has won and a new age, a Utopia is upon us.”

Language and System Design Philosophy

Smalltalk wasn’t originally a full language, but a live object system:

“Small Talk was never a programming language it was a programming system… there is no way to express a class as a linguistic construct.”

Early Smalltalk used a clever hack for class definition:

“Yes a class just happens to be this object that knows how to make other objects… but it is a hack and it's important to recognize that.”

Textual syntax was later added to enable integration with tools like source control:

“We came up with an actual textual Syntax for Smalltalk… interacting with Source Control Systems became really easy.”

Resolving Dialect & Library Fragmentation

Overcoming dialect fragmentation:

“We realized all this and came together to define a Common Language.”

Library standardization was harder but necessary:

“The situation with the libraries was a bit of a mess… but we realized that the benefits of having a standard… outweighed [vendor-specific advantages].”

Reflection and Mirrors

Original reflection API mixed base and meta levels:

“These class objects… are playing two roles… that architecture was replaced with mirrors.”

Mirrors cleanly separate base-level and meta-level:

“We clearly separated the base level and the meta level… that has all kinds of advantages for deployment, distribution, security.”

Mobile and UI Integration

Early design enabled seamless remote mobile dev:

“We were already running images that could manipulate other images… fantastic development experiences almost immediately.”

Binding to native UI was essential for adoption:

“We realized early on that we should… bind to Native stuff… they run natively on all the platforms.”

Tool evolution toward navigation-based UIs:

“We wouldn’t dream of using the 40-year-old design of a browser… we evolved our tools.”

Foreign Function Interfaces and “Alien” Objects

Unified model for system integration:

“These things outside of Smalltalk… they're second class objects… but they were objects.”

Alien objects replaced clunky primitive syntax:

“We didn’t need these hacky things like the Primitive syntax… everything was an object.”

Modularity and Deployment

Object ecosystems are difficult to transfer without structure:

“Fish and objects live in an environment… if you want the fish to be transferred and survive you have to bring its friends.”

Deployment as serialized object graphs:

“We realized that an application… was also an object… we just had to serialize it.”

Avoided “extraneous concepts like packages”:

“We used the concepts of classes and objects to solve our distribution and modularity problems.”

Optional Typing and Pluggable Type Systems

Types as useful but not mandatory:

“Types can be useful at times… we certainly don’t want them telling us how to live.”

Static analyses used for optimization:

“Extra information helps… we could have multiple analyses that didn’t conflict.”

Web and Live Environments

Ahead of the curve with web-based IDEs:

“We built complete programming environments with all their features running in the browser long before anyone else.”

True liveness beyond class browsers:

“We never want to look at Dead code… environments evolved to show Exemplar values.”

Integration of ML into the environment:

“We started to incorporate [ML] in our live programming environments.”

Performance and JIT

Ahead of AOT compilation trend:

“We had systems that would… have a database of compiled methods… ready to apply them instantly on Startup.”

Addressed Apple’s no-JIT constraint early:

“We already had techniques for that… just deploy that and turn off the jit.”

Security through Object Capabilities

Reinvention of capability-based security:

“The capability you want is an object… the only damage you can do is through a message send.”

Enforced encapsulation and access control:

“We needed an access control model… public, protected, and private messages.”

Education and Longevity

Revolutionary ideas in education:

“Smalltalk had these ideals of Education from day one.”

Avoided student drop-off by staying dominant:

“We short circuited [students leaving] because of all our previous successes.”

Hypertext-based live documentation:

“We could put live widgets embedded in text long before the worldwide web.”

Satirical Punchline

Entire talk is satirical utopia:

“This is the world as it should be… imagine if the world wasn’t like the way I described it…”

Ironic jab at real-world language use:

“Think if one of the world’s largest… sites ran on PHP… we all know Facebook runs on Smalltalk.”

• Core Concept: Missionary is a Clojure library for supervised data flow programming, providing a universal language for handling asynchronous event sources in both frontend and backend applications.

  "the missionary is a closure library for supervised data flow programming so this is a common language to manipulate asynchronous event sources um that you can use from anywhere in the stack"

• Problem Addressed: The library tackles the difficulty of resource management in concurrent programming, where resources (like circuits, UI components, or threads) have a life cycle that must be explicitly managed.

  "this is in my "pinion the main reason why concurrency is hard a resource is an object with a Time dimension because it has a Time Dimension it also has a life cycle"

• Limitations of Garbage Collection: Traditional garbage collection is insufficient for this problem due to bidirectional references between producers and consumers, leaving the producer unaware if a resource is still needed.

  "garbage collection doesn't help the reason why it doesn't fa is because um the reference to the resource is B directional there is it is referenced by the consumer and it is also referenced by the producer"

• Proposed Solution: The solution is explicit structure, which enables automatic, demand-driven resource management by binding the resource's allocation lifespan to the period when its data is actually required.

  "what we want to achieve is to bind um the time span where the resource is allocated to the time span when the data produced by this resource is actually needed so it's demand WR resource maintenance"

• DAG Supervision for Shared Resources: For complex scenarios with shared resources, missionary moves from a simple tree supervision model to a Directed Acyclic Graph (DAG) supervision model, where a resource can have multiple supervisors.

  "so now the supervision uh three is not a three anymore it's a dag has a shared resource here"

• Core Supervision Policy: The policy for managing shared resources is "allocate on first use and dispose on last," elegantly summarized by the real-world analogy of "the last one turns of the lights."

  "the last one turns of the lights uh so this is a dck for the real world"

• Improving on Functional Effect Systems: Missionary was conceived to improve upon functional effect systems like Rx by properly implementing continuous time, which requires synchronicity semantics to avoid event-ordering issues.

  "the IDE was uh continuous time is a good idea...the problem is um well at this time the popular FX system in Java was RX and RX doesn't properly Implement continuous time uh so the reason why it doesn't properly Implement continuous time is because it has no synchronicity semantics"

• Glitch-Free Propagation: A key innovation is a bidirectional flow protocol and a propagation algorithm that uses a priority queue to traverse the dependency graph, ensuring atomic updates and preventing inconsistent intermediate states, known as "glitches."

  "to solve the glitch problem...the string protocol doesn't support that and so therefore the idea is can we have our and El two"

• Bidirectional Flow Protocol: The protocol allows a producer to signal that its state is invalidated before a new value is computed, enabling a two-phase process of invalidating dependent nodes and then recomputing them in the correct order.

  "bidirectional IPC is what we need to implement uh back pressure and solve the git problem that I talk just before"

• Successful Validation: The model has been battle-tested at scale through its use in Electric Clojure for serious business applications, proving its robustness and success in managing complex, dynamic, and massively concurrent UIs.

  "that's a success we've been validating that model at scale with electric closure we we've been working on that for more than two years now uh we've been using electric closure for serous business application and this algorithm just well we we experience really no problem with that"

• Core Thesis: Functional programming is an excellent fit for creating reactive or "situated programs" that must continuously interact with their environment, contrary to some earlier views.

  "we're going to use functional programming to make situated programs um i'm going to show you that rich is wrong about it functional programming is actually a very good fit for situated programs and we're going to see why."

• Functional Effect Systems: The fundamental principle is to describe computations as values rather than executing them immediately. An effect is a description of an action to be performed.

  "an effect is a description of something to be done that's the essence of functional programming instead of doing we describe."

• Core Operators: The system is built on operators like pure for turning any value into an effect and bind for composing effects sequentially.

  "pure takes an arbitrary value turns that into an effect... blind is about sequential composition so we want to do something and then do something else."

• Concurrency and Supervision Trees: Concurrent operations are managed within a process tree. This structure is critical for handling failures and managing resources. When one parallel process fails, its siblings must be canceled to prevent wasted resources.

  "what we get is not a list of process it's a tree of processes and that's that's very important... when it happens you want the error to be processed by another process and this process is the supervisor."

• Structured Concurrency: Functional programming naturally enforces a well-defined supervision hierarchy, where combinators also define the supervision strategy, preventing orphaned processes.

  "in functional programming it's impossible to do that so you get structural concurrency by default that is you are forced to build your program in such a way that the supervision trees is properly structured."

• Streams vs. Signals: The talk distinguishes between two types of long-lived effects:

• Event Streams: A discrete series of events where each event is critical and must be processed. They require backpressure to prevent data loss. > "an event stream is not defined when events don't happen it's discrete time... losing an event is really bad... to represent stream's effect the effect representation must implement by pressure."

• Signals: A continuous value representing the state of an identity (e.g., mouse position). They are always defined, only the latest value matters, and they benefit from lazy sampling. > "the signals represent the state of an identity... at any point in time you can take a snapshot... only the latest value matters... if you want to represent signals as effects you won't play something."

• Missionary Library: A Clojure library that implements these concepts, providing operators for two effect types: tasks (for single values) and flows (for multiple values).

  "this is mission array it's a closure library that works enclosure and closure script and it's a collection of purely functional operators that work on effects and there's two kinds of effects there is tasks and flows."

• Language Extensions: Missionary avoids "callback hell" through language extensions that provide a functional version of async/await, allowing for more readable, sequential-looking code that is internally transformed into callbacks.

  "the solution is to extend the language so we we extend closure with another operator that's the idea of async await but now it works in functional effects."

• Awaiting Flows: A powerful and unique feature is the ability to "await" a flow (a stream of multiple values). This reruns a computation for each new value while automatically canceling the previous computation if it's still running.

  "if we get a new value of the state and the previous value is still being computed we want to interrupt this this previous computation and start the new one you we are only interested in the latest value so we want to discard the previous computation."

• Key Differentiators: Missionary stands out due to its powerful language extensions (an expressive alternative to monads) and its first-class support for both discrete-time (streams) and continuous-time (signals) effects.

  "what makes it unique is language extensions uh it's alternative to monad it's as much as powerful but it's much more expressive... discrete versus continuous time we need both."

• Computational Challenges:

  • We lack effective computational methods: "i think we actually have the foggiest idea how to compute very well."
  • Importance of fast, efficient processes: "it took only 100 milliseconds... we don't understand how to do."

• Genomic Complexity:

  • Human genome's complexity and flexibility: "with a small change to it you make a cow instead of a person."
  • High-level language for biological processes is unknown: "what I'm interested in is the high-level language that's necessary to do things like that."

• Programming Evolution:

  • Legacy of programming assumptions based on scarcity: "all of our sort of intuitions from being programmers have come from a time of assuming a kind of scarcity."
  • Current abundance of resources shifts the focus: "memory is free, computing is free."

• Security and Correctness:

  • Traditional concerns of correctness and security are secondary: "people worry about correctness... is it the real problem? maybe... most things don't have to work."
  • Evolution and adaptability of code are crucial: "we spend all our time modifying existing code."

• Programming Constraints:

  • Early decisions in programming constrain future changes: "we make decisions early in some process that spread all over our system."
  • Need for flexibility in modifying systems: "organize systems so that the consequences of decisions we make are not expensive to change."

• Generic Operators and Extensions:

  • Dynamically extensible operations: "dynamically extend things while my program is running."
  • Symbolic algebra as an extension of arithmetic: "expand this arithmetic on functions... it's a classical thing people can do in algebra."

• Propagators and Parallelism:

  • Concept of propagators for parallel computation: "propagators are independent little stateful machines."
  • Parallelism and monotonic information merging: "we don't actually put values in these cells we put information about a value in a cell."

• Truth Maintenance Systems (TMS):

  • Maintaining and improving data consistency: "truth maintenance systems... maintain the best estimates of what's going on."
  • Dependency-directed backtracking for efficient problem-solving: "automatically find for me the consistent sub consistent sub the sub world views that are consistent."

• Historical and Educational Insights:

  • Historical evolution of computation: "when I started computing in 1961... the total amount of memory is probably about 10 kilobytes."
  • Educational gaps between theory and practical engineering: "what we taught the students wasn't at all what the students actually were expected to learn."

• Vision for the Future:

  • Future computing systems must be inherently parallel, redundant, and flexible: "future... computers are so cheap and so easy to make... they can talk to each other and do useful things."
  • Importance of evolving current computational thinking: "we have to throw away our current ways of thinking if we ever expect to solve these problems."

• Summary and Call to Action:

  • Main challenge is evolvability, not correctness: "problem facing us as computer engineers is not correctness it's evolvability."
  • Proposals include extensible operations and new architectural paradigms: "extensible generic operations... a more radical proposal is maybe there are freedoms that we can unlock by throwing away our idea of architecture."

Bet on context engineering for rapid iteration.

“Manus would bet on context engineering. This allows us to ship improvements in hours instead of weeks, and kept our product orthogonal to the underlying models: If model progress is the rising tide, we want Manus to be the boat, not the pillar stuck to the seabed.”

Design around the KV-cache to optimize latency and cost.

“If I had to choose just one metric, I'd argue that the KV-cache hit rate is the single most important metric for a production-stage AI agent.”

Mask tool availability rather than removing tools mid-iteration.

“Rather than removing tools, it masks the token logits during decoding to prevent (or enforce) the selection of certain actions based on the current context.”

Use the file system as scalable, persistent context.

“That's why we treat the file system as the ultimate context in Manus: unlimited in size, persistent by nature, and directly operable by the agent itself.”

Manipulate attention through recitation of objectives.

“By constantly rewriting the todo list, Manus is reciting its objectives into the end of the context. This pushes the global plan into the model's recent attention span, avoiding ‘lost-in-the-middle’ issues.”

Keep errors in the context to enable adaptive behavior.

“One of the most effective ways to improve agent behavior is deceptively simple: leave the wrong turns in the context. When the model sees a failed action—and the resulting observation or stack trace—it implicitly updates its internal beliefs.”

Avoid over-few-shotting by injecting controlled diversity.

“The fix is to increase diversity. Manus introduces small amounts of structured variation in actions and observations—different serialization templates, alternate phrasing, minor noise in order or formatting.”

Context engineering defines agent performance and scalability.

“Context engineering is still an emerging science—but for agent systems, it's already essential. … Engineer them well.”

• Abstractions are based on assumptions

• We assume some properties which allows us to neglect them.

• The more we assume the simpler our abstraction.
• The less we assume, the more we need to address, and the more complex our abstraction becomes.

• Summary of Tech Talk on Software Abstractions and Model Assumptions

• Overview of Book on Closure: The speaker discusses a book written to serve as a comprehensive second book on Closure, focusing on choosing the appropriate parts of the language based on different programming situations.

• "the goal of this book was to be the best second book that you could read about closure where you already know what you can do with the language but you don't know why you would use one part of the language versus another to solve a particular problem."

• Importance of Names in Programming: Emphasizes the significance of names in software, noting the lack of deep discussion in education and practice, leading to debates often decided by authority rather than merit.

• "the first chapter that I wrote was about names and for all that we pay a lot of lip service to names being one of the two hard problems in software we don't really spend a lot of time confronting them head-on."

• Challenges in Defining "Abstractions": Discusses the difficulty in defining and writing about abstractions in software, leading to extensive research without writing progress.

• "things were going well until I tried to do the same for abstractions right what makes an abstraction good what makes it bad if it's bad how do we make it better."

• Different Interpretations of "Abstraction": Highlights the two distinct concepts the term "abstraction" covers in computer science, demonstrated by church numerals and cons cells in LISP.

• "we use the word abstraction to mean two fairly distinct concepts and I think this is demonstrated by two ideas that are very much at the heart of lists which are church numerals and cons cells."

• Timelessness vs. Practicality in Abstractions: Examines how some abstractions remain relevant due to their theoretical utility, while others fall out of favor due to practical inefficiencies in changing environments.

• "the difference between these things is not what they are but how we judge them one of them is timeless right it is judged against a standard which does not change but the other one is judged against lis very rapidly changing sort of standard."

• Role of Environment in Software Abstractions: Argues that understanding the environment and its assumptions is crucial in defining and evaluating abstractions.

• "when we define an abstraction specifically a software abstraction we have to have three parts we have to have the model which is the thing that we implement we have to have the interface which is a means of interacting with that model and we need to have the environment which is everything else."

• Adaptability and Principle in Software Systems: Proposes that effective software systems combine principled and adaptable components, ensuring robustness and flexibility.

• "if we have a sort of an adaptable abstraction that contains as little principal pieces we can see that change comes from the outside and as change comes in from the outside these principled fragile pieces are protected."

• Complex Adaptive Systems and Software: Suggests modeling software systems on complex adaptive systems to manage change effectively through principled and adaptable components.

• "these sorts of systems right where you have sort of an adaptable whole and principled subcomponents exist at every level of the world this is empirically a very successful strategy."

• Conclusion and Encouragement to Engage with Concepts: Concludes by encouraging the audience to read the book and engage in discussions about the concepts, highlighting the evolving understanding of software abstractions.

• "if this was interesting to you I really encourage you to read my book I think that it's much more clearly articulated there than it has been on this stage but if you do read it right I really encourage you to talk to me about it."

• Introduction to Queues:
• "Queues really are pretty simple, right? You have a producer that enqueues messages, you have a consumer that dequeues messages."
• "Something that is maybe non-obvious is that in addition to the act of enqueuing being a side effect... we get back to just confirmation that it was added."

• "Queues are a way of dealing with actions... we of course are in the business of making computers do things."

• Core.async Channels and Queue Mechanisms:

• "A core.async channel has not one but three queues: a buffer that holds the messages, a puts queue where producers that are trying to add to a full buffer will wait, and a takes queue where consumers that are trying to take messages from an empty buffer will wait."

• "The blocking queue under the covers looks a great deal like core.async... it has a buffer that holds the messages... it’s just a place for threads to park themselves."

• Ubiquity and Importance of Understanding Queues:

• "Queues are ubiquitous; they’re everywhere and it sort of behooves us on some level to understand them."

• "The reason that queues are so ubiquitous is because they separate what we want to have happen from when it happens."

• Closed vs. Open Systems:

• "A key distinction in queuing theory is between closed and open systems."
• "A closed system is where as we produce something, we must wait for the consumer to complete it before producing something else."

• "Many of the systems that we build are open systems where there is no coordination... requests are processed as they come in, often before we’re ready."

• Simulation of Queue Behaviors:

• "Tasks are arriving according to this exponential distribution... the complexity of the tasks is not very well modeled by an exponential distribution."

• "As the system becomes unstable... it just sort of takes off like a rocket... what happens when a queue grows out of control."

• Dealing with Queue Overload:

• "Unbounded queues are fundamentally broken because it puts the correctness of your system in somebody else’s hands."

• "We cannot have our strategy for dealing with too much data be to hold onto it and hope for the best."

• Strategies for Managing Excess Data:

• "When too much data shows up, we have only three strategies: drop the data on the floor, reject the data, or pause the data."
• "Dropping the data is valid for problems where newer data makes older data obsolete."
• "Rejecting the data... is what you see when you go to an overloaded website and it returns a 503."

• "Pausing the data or exerting back pressure... is often the most neutral choice we can make."

• Benefits of Buffers and Back Pressure:

• "Buffers allow us to make sure that our throughput is more stable and going at a higher rate."

• "Back pressure enacting and retracting is not free... buffers help with the overall throughput at the expense of latency."

• Best Practices for Queue Management:

• "Always plan for too much data... using back pressure wherever possible."
• "Buffers are useful for constant throughput but should be used where absolutely necessary."

• "Avoid composing buffered queues wherever possible as each additional buffer magnifies the results of the next."

• Application Example: Durable Queue and S3 Journal:

• "I end up writing two libraries, one of which is called durable queue, a disk-backed queue, and another called S3 Journal, built on top of the durable queue."

• "The system looks like this: there are two stages, the HTTP server concerned with persisting data and another loop uploading it to S3."

• Ensuring System Stability:

• "Measure what is the health of our system... quantify how much we might have lost when a machine goes down."

• "It’s really crucial when building these systems to understand what it means to complete something."

• Monitoring and Instrumentation:

• "We have metrics now... gives us a little bit more visibility in the system."

• "Be picky about your tools... prefer something which actually tells you how fast it is."

• Final Recommendations:

• "Unbounded queues are bounded by memory or some fixed shared resource."
• "Account for what happens when your system receives too much data."
• "Use back pressure to defer the choice to someone who understands the application better."
• "Demand metrics everywhere to ensure system robustness."

This summary outlines the core arguments of the paper "Biology as Reactivity," which posits that biological systems, particularly living cells, can be understood as the ultimate reactive systems. * Core Thesis: The principles and tools used for analyzing reactive systems in computer science can be applied to biological research to create virtual experimentation environments.

"Concepts, languages, and tools for the description and analysis of reactive systems can help in the process of biological discovery, ultimately by providing biologists with virtual experimentation environments."

• The Cell as a Reactive System: A living cell is presented as a quintessential reactive system, where its behavior is a function of not just the inputs, but also their timing, order, and location.

  "A living cell, we claim, is not only reactive in nature, but is the ultimate example of a reactive system, and so are collections thereof."

• Adaptivity as a Form of Reactivity: The adaptive nature of biological systems to both internal and external stimuli is a key aspect of their reactivity.

  "Biological systems are also highly adaptive, to both internal and external changes; they use signals coming in from receptors and sensors, as well as emergent properties to fine-tune their functioning. This adaptivity is another facet of the reactivity of such systems."

• Key Concepts in Biological Reactivity: The understanding of biological systems as reactive necessitates a grasp of fundamental concepts like parallelism, interaction, and causality.

  "Hand-in-hand with the central notion of reactivity go the discrete event-based execution and simulation of dynamical systems, which requires a fundamental understanding of parallelism, interaction, and causality; the design of complex systems from building blocks, requiring means for composition and encapsulation"

• Signal Propagation and distributed control: The paper gives the example of muscle contraction to illustrate how signals are propagated in a distributed manner.

  "But notice that when a muscle contracts, the nerve reaches only some of the muscle cells, and responsibility for the remainder of the signal's effect must be taken over by other mechanisms."

• A Novel Programming Paradigm: Behavioral programming is a new way to create reactive systems by focusing on how the system should behave, which aligns with the initial requirements.

  "We describe an implementation-independent programming paradigm, behavioral programming, which allows programmers to build executable reactive systems from specifications of behavior that are aligned with the requirements."

• Modular and Flexible Development: It allows for the addition and modification of existing behaviors through software modules, which simplifies dealing with incomplete or conflicting requirements.

  "Behavioral programming simplifies the task of dealing with under-specification and conflicting requirements by enabling the addition of software modules that can not only add to but also modify existing behaviors."

• Scenario-Based Programming: Behavioral programming originated from the scenario-based language of live sequence charts (LSC), which allows for creating executable specifications of reactive systems.

  "The work on behavioral programming began with scenario-based programming, a way to create executable specifications of reactive systems, introduced through the language of live sequence charts (LSC) and its Play-Engine implementation."

• Core Programming Idioms: The paradigm uses specific idioms to express what must, may, or must not happen, similar to modal verbs in natural language.

  "Reminiscent of modal verbs in a natural language (such as shall, can or mustn't), they state not only what must be done (and how) as in standard programming, but also what may be done, and, more uniquely to behavioral programming, what is forbidden and therefore must not be done."

• Event-Driven Approach: To begin development, a common set of events relevant to the system's scenarios is determined.

  "In behavioral programming, all one has to do in order to start developing and experimenting with scenarios that will later constitute the final system, is to determine the common set of events that are relevant to these scenarios."

• Conflict Resolution and Scalability: The paradigm includes composition operators to manage and resolve conflicting behaviors, analysis tools like model checkers, and architectures for large-scale applications.

  "We deal with these issues by using composition operators that allow both adding and forbidding behaviors, analysis tools such as model checkers, and architectures for large-scale applications."

• Language Independence: The principles of behavioral programming can be implemented in various programming languages and environments.

  "Behavioral programming principles can be readily implemented as part of different languages and programming approaches, with possible variations of idioms."

To my understanding, the talk started with the following premise:

Global state is bad, DB is global state => DB is bad (use Rama instead)

I'd argue that: every observable state is global state

• Global state refers to a state that is outside of some context but still affects it.
  • Affects it means that the state is observable in some way from outside the state's context.
• If we consider that when we talk about the problems of state we effectively means observable state and the issues it causes outside its context.
• Then by acknowledging it has effects on other contexts outside its own context we acknowledging that it is a global state (because it affects some context from outside).

So after establishing every observable state should be considered global state to some extent (of some contexts). The way to solve this problem is eliminate state. Can this be done? I don't know (I don't believe it can)

But more practically: did Rama eliminated state? No, it didn't, if you append an event to an event log you have changed the state of that event log and potentially every view that was derived from that event log.

So:

global state is bad, db is global state, rama is global state => everything is bad

https://stopa.io/post/296

Certainly! Here’s a summary and key insights from the article "A Graph-Based Firebase" by Ivan Stopa.

Summary: "A Graph-Based Firebase"

The Problem

Modern, delightful apps like Figma, Notion, and Linear share three standout features: - Optimistic updates (instant UI feedback) - Multiplayer collaboration - Offline mode

Building these features is hard. Most frameworks and databases don’t make it easy-especially when you want all three together.

Existing Solutions & Their Limits

• Firebase: Great at optimistic updates, offline mode, and reactivity. But it falls short for complex data relations and permissions.
• Supabase & Hasura: Support relations (thanks to SQL/Postgres), but lack robust local/offline abstractions and have limited permission systems for complex apps.
• Custom Architectures: Apps like Figma/Notion/Linear end up building their own stack: normalized in-memory stores, custom permission layers, manual syncing, and more.

The Core Insight

All these apps end up re-inventing a kind of database, but one that works both locally (in the browser) and remotely (on the backend), with: - Graph/relational queries - Expressive permissions - Built-in reactivity, offline, and optimistic updates

The Proposed Solution: Instant

Ivan and his co-founder built Instant: a graph-based, Firebase-like platform with: - Triple store backend (instead of SQL), enabling flexible relations and recursive queries. - Datalog-inspired query language (instead of SQL or GraphQL), for nested, relational data. - Unified local/remote architecture: The same query/mutation language runs in the browser and on the server. - Expressive, Facebook-like permissions: Functions for allow/deny rules, not just boolean expressions or row-level policies. - Lightweight and easy to start: No heavy schemas or large client libraries.

Why Not SQL?

• SQL is heavyweight (1700+ page spec), not optimized for nested/recursive queries common in UIs.
• Frontend needs nested data, but SQL returns flat rows-requiring GROUP BY, JSON hacks, or complex post-processing.
• Implementing full SQL reactivity and sync is overkill for most app needs.

The Vision

A new abstraction-something like a graph database in the browser-could make it dramatically easier to build fast, collaborative, offline-capable apps. If the local and backend database speak the same language, you get: - Real-time sync - Optimistic updates - Offline support - Powerful, maintainable permissions - Simple, nested queries for UI

Key Takeaways

• Modern app UX is a database problem: The hardest parts of delightful apps (speed, collaboration, offline) are really about how data is managed and synchronized.
• Existing tools force trade-offs: You can have relations or offline support, but not both. You can have permissions or reactivity, but not both.
• A new kind of database abstraction is needed: One that combines the best of Firebase (local-first, reactivity) with the best of SQL/Graph (relations, permissions).
• Triple store + Datalog is promising: It enables flexible, nested queries and is easier to make reactive and local-first than SQL.

If You’re a Developer…

• If you’ve struggled to build fast, collaborative, offline-friendly apps, you’re not alone.
• Watch for tools like Instant (or similar local-first, graph-based databases).
• The future of app development may look a lot more like building with a distributed, reactive, permissioned graph database-on both client and server.

Let me know if you want a deeper technical breakdown, a comparison table, or actionable recommendations for your own projects!

Citations: [1] A Graph-Based Firebase https://stopa.io/post/296 [2] A Graph-Based Firebase https://stopa.io/post/296

• Definition and Nature of Design:

  • Design involves preparing executable plans and structuring form.

    "Prepare the plans for a work to be executed... plan the form and the structure of that thing." - Effective design separates elements to enable later composition.

    "Designing is fundamentally about taking things apart... in such a way that they can be put back together."

• Misconceptions About Design:

  • Good design isn't mere documentation generated post-implementation.

    "Can we just write our program and then generate some documentation?... No, that's not a plan." - Bad past experiences with monolithic designs don't invalidate design itself.

    "Those were plans but those are not good plans... doesn't mean planning is bad."

• Key Elements of Good Design:

  • Identify underlying problems rather than stated needs or symptoms.

    "Decompose those wants and needs into problems." - Differentiate known from unknown requirements, causes from symptoms.

    "Take apart causes and root causes from symptoms." - Address unstated, implicit requirements (e.g., maintainability, performance).

    "Unstated requirements... problems nobody wants in the future." - Separate concerns clearly: time/order, places/participants, information/mechanisms.

    "Separating apart when things happen... places and participants... information our systems manipulate."

• Iterative and Decompositional Nature of Design:

  • Design involves continuous decomposition and recomposition, supporting iterative changes.

    "Good design process is iterative... breaking things down until they're nearly atomic."

• Why Good Design Matters:

  • Design aids understanding, coordination, extension, reuse, and efficient testing.

    "Design helps you understand a system... coordinate... extension... reuse... design-driven testing." - Design leads to greater efficiency by simplifying iterative changes.

    "Easier to iterate a design than... an implementation."

• Composition and Performance Analogies (Bartók and Coltrane):

  • Composers set constraints deliberately, solving self-created problems through design.

    "First thing composers do... they create their own problems to solve." - Bartók (composer) specifies details precisely; Coltrane (performer) dynamically applies studied compositional knowledge.

    "Bartók completely specified every note... Coltrane had melody and changes... providing constraints for the performer."

• Improvisation vs. Planning in Software:

  • Improvisation isn’t random; it relies heavily on preparation and deep understanding.

    "Improvisation is not spontaneous emoting... application of knowledge and vocabulary." - Good developers, like good improvisers, combine studied preparation and dynamic execution.

    "Coltrane was working out this composition dynamically... resources behind it were things he had prepared."

• Harmony as Critical Design Skill:

  • Harmony is about congruity and simultaneous fitting together.

    "Harmonic sensibility is of critical design skill... systems that fit together." - Bartók and Coltrane exemplify mastery and innovation in harmonic principles.

    "Masters of harmony... retained focus on what fits together."

• Programming Languages as Instruments:

  • Programming languages/tools should resemble instruments: simple, focused, and designed for skilled users.

    "Languages and libraries like instruments... minimal yet sufficient." - Instruments are intentionally limited, enabling deep expertise through practice.

    "Piano can’t play in-between notes... saxophone one note at a time... instruments are minimal yet sufficient."

• Problems of Excessive Complexity and Choice:

  • Too many options or excessive configurability leads to poor design, overwhelming users.

    "Having a lot of choices... opposite of enabling us to accomplish things... constraint is a driver of creativity." - Simplicity and constraint enhance usability and creativity.

    "Every module is a good idea... adding them together you end up with something you can't play."

• Tools Should Empower Skilled Users, Not Beginners:

  • Software tools/instruments should not overly cater to beginners, sacrificing depth.

    "Instruments are made for people who can play them... beginners can't play yet." - Effort and practice are necessary for mastery; attempts to eliminate all effort weaken tool quality.

    "Effort is not a bad thing... neither learning nor teaching is effort-free."

• Design as Decision-Making:

  • Effective design clearly communicates deliberate, beneficial decisions.

    "Design is about making decisions... conveying those decisions to the next person." - Failure to decide clearly (excess configurability) is a design failure.

    "If you make everything configurable you're failing to design."

• Summary Recommendations:

  • Embrace decomposition and iterative refinement in design.
  • Develop harmonic sensibility for building coherent, maintainable systems.
  • Create simple, constrained, and purposeful software tools/instruments.
  • Prioritize deliberate decision-making to convey clarity and utility.

• Final Takeaways:

  • "Design like Bartók" by applying meticulous compositional care across scales.
  • "Code like Coltrane" by dynamically applying studied design knowledge.

  • Seek simplicity and harmony, avoiding unnecessary complexity and choice.

    "Take things apart... design like Bartók... code like Coltrane... pursue harmony."

Overview & Motivation

Repeatedly failed to write a post, realizing it should be a talk:

“It turns out that I wasn’t really writing a post; I was actually preparing a talk.”

Central Topic: React Server Components and distributed computations between two machines using React concepts.

“It’s about everyone’s favorite topic, React Server Components.”

Act 1: Recipes (Imperative) vs. Blueprints (Declarative)

Tags vs. Function Calls:

Visual and structural differences:

“< and > are hard and spiky and ( and ) are soft and round.”

Similarities:

Both reference named operations (functions or tags) and accept arguments.

Both allow nesting.

“Clearly, function calls and tags are very similar...they let us elaborate by nesting further.”

Differences:

Tags (declarative):

Often nouns; represent timeless structures (blueprints).

Convenient for deep nesting, clearly marking structure.

Time-independent, passive descriptions.

“Tags tend to be nouns rather than verbs... nouns are easier to decompose.”

Function calls (imperative):

Often verbs; represent sequential actions (recipes).

Execution order critical.

“A recipe prescribes a sequence of steps to be performed in order.”

Remote Procedure Calls (RPC) and Async/Await

Problem: Calling Functions Across Computers

RPC concept introduced: Functions across network boundaries.

async/await: Simplifies asynchronous calls but still has limitations (coupling, losing direct references).

“An async function...may pause execution...async and await propagate upwards.”

Import RPC idea: Extends importing to remote function calls while maintaining references and type-checking.

“Let’s invent a special syntax...import rpc because what we’ve described here has been known for decades as RPC.”

Potential Calls (Tags as Deferred RPCs)

"Potential function calls": Represented by tags; calls that might happen in the future.

“It’s a blueprint of a function call.”

Nested tags: Express dependencies naturally.

“Dependencies between potential calls...should be expressed by embedding these calls inside each other.”

Splitting Computation in Time and Space

Computation split in time: Returning partial functions that capture necessary data (closures).

Computation split across space (client-server): Splitting execution between two computers, handling data passing explicitly.

“It’s an interesting shape—a program returning the rest of itself...closure over the network.”

Two Types of Operations: Components vs. Primitives

Components (Capitalized): "Brains" of a program; flexible, timeless, and declarative, embedding tags without introspection.

“Components are truly timeless...they accept tags as arguments.”

Primitives (lowercase): "Muscles"; introspect arguments, execution order sensitive, imperative, execute last.

“Primitives introspect arguments...they must know all their arguments.”

Execution Phases:

1. Interpret (thinking): Processes Components freely without strict order.

2. Perform (doing): Executes Primitives strictly inside-out.

“First, you need to think...then you need to do.”

Act 2: Reflections and Dialog

Meta-dialog: Reflection on the writing process itself; writer and reader dialogue, acknowledging uncertainty and experimental nature of content.

“The Writer: I have a rough idea, but truthfully, I’m pretty much winging it.”

Core Conceptual Innovations

Tags as code/data pairs: Potential function calls represented explicitly as data (tags), allowing deferred execution across contexts.

Program as distributed computation: A single conceptual function spanning multiple runtime environments (Early and Late worlds).

Timelessness and Flexibility: Components allow arbitrary computation ordering; Primitives enforce execution order.

Key Quotes & Ideas:

Blueprints vs. Recipes:

“A blueprint describes what nouns a thing is made of...a recipe prescribes a sequence of steps to be performed.”

RPC and Potential Calls:

“A tag is like a function call but passive, inert, open to interpretation.”

Components and Primitives Separation:

“Components are the ‘brains’...Primitives are the ‘muscles’.”

Importance of Introspection vs. Embedding:

“If a function only embeds an argument without introspection, you can delay computing it.”

Conclusion (Conceptual Breakthroughs)

Distributed React Model: Redefining client-server interaction as React component structures.

Future implications: Suggests moving common primitives into lower-level implementations to optimize distributed computation.

“If many programs used the same Primitives...move their implementation to Rust or C++.”

Talk overview: Discusses integrating Resource Description Framework (RDF) with language models (LMs), aiming to leverage RDF’s logical precision and LM’s linguistic flexibility.

"We'll go over what rdf is, what language models are, and why they go well together."

Controversial statements: Presents two conflicting viewpoints:

LMs are valuable technological advances with profound philosophical implications.

"Language models are pretty neat and I like them... We never imagined we'd be able to compute over unstructured text like this."

AI/LMs have negative societal impacts, including environmental harm, disinformation, and devaluing art and labor.

"Language models suck or rather AI sucks... It's bad for the environment, art, labor... we're drowning in slop and spam."

Personal perspective and approach: Advocates mindful, responsible tech use; avoid superficial social media debates; focus on enhancing tech to achieve net positive societal impact.

"Be mindful of the impact... don't talk about this stuff on social media... how good does it have to be before it becomes net good?"

Introduction to RDF: RDF, developed post-internet by symbolic AI enthusiasts (W3C), is designed for precise, abstract knowledge representation via "triples" (subject-predicate-object).

"RDF is an attempt to tackle some of the hardest problems of knowledge representation and reasoning."

Critiques of RDF: Historically misunderstood due to complex formats (RDF XML), overly complex object-oriented libraries, and failed Semantic Web hype.

"RDF XML... was a verbose complex format... The Semantic Web was simply overhyped."

Strengths of RDF: Despite critiques, RDF remains robust and is heavily adopted in scientific, governmental, and enterprise environments for precise information modeling.

"RDF is being used productively in science, heavy industry, government."

Core components of RDF:

Resources: Unique identifiers (IRIs) representing any real or abstract entities.

"Anything that can be the subject of language can be the subject of rdf."

Triples: Fundamental units of RDF, expressing precise semantic relationships clearly.

"Subject predicate object... one of the most granular ways of representing information."

Importance of context in RDF: RDF IRIs inherently carry context, resolving linguistic ambiguity through precise identifiers.

"The goal of IRIs in RDF is that they carry their context with them."

RDF as knowledge representation: RDF is more than data storage; it’s structured for logical entailment, inference, and reasoning capabilities, akin to symbolic AI.

"RDF... is about representing knowledge... it's designed to make it possible to talk about all the things I know."

Introduction to Language Models (LMs): Originating with Transformers (Attention Is All You Need, 2017), LMs revolutionized natural language processing by capturing grammar, syntax, semantics, pragmatics, and reasoning patterns from vast datasets.

"Attention is all you need... unlocks everything language models can do today."

Defining LMs: LMs are pure mathematical functions trained on language, predicting next tokens based on vast statistical measurements (latent semantic space).

"Language model... is a pure function that predicts the next token."

Current state of LM programming: Due to the complexity and opaque outputs of LMs, effective usage relies on controlling inputs through prompt engineering, information retrieval, query generation, tool use, and agent design.

"All AI programming... is altering the inputs... Prompt engineering... Tool use... Agents."

RDF and LM integration: RDF uniquely bridges structured data (logic) with natural language, enabling precise interactions (querying, reasoning) between LMs and structured data via RDF's semantic precision and natural linguistic alignment.

"We should be putting rdf data in our prompts... RDF is really surprisingly good at going back and forth between natural language and rdf."

Advantages of combining RDF and LMs:

Simplifies querying structured data through natural language, enabling reasoning and inference beyond simple SQL queries.

"If my RDF implementation supports reasoning... all that complexity is abstracted out of the model."

Allows soft inference using LMs to approximate implicit knowledge, overcoming limitations of strictly rule-based AI systems.

"Soft inference... we now have hard inference... and soft inference... you can do reasoning you couldn’t do with either alone."

Neurosymbolic AI: Integrating RDF (symbolic reasoning) with LMs (neural networks) represents neurosymbolic AI, combining the strengths of precise logic with flexible language modeling.

"This is the central insight behind what's called neurosymbolic AI."

Practical vision and conclusion: Advocates using AI/RDF to automate tedious "data chores," proposing a constructive, pragmatic use-case-driven vision of AI tech.

"I do a lot of programming is dishes and laundry of data... trying to build it... we're going to bring back the Semantic Web with AI."

Figma's blog post, "How Figma's Multiplayer Technology Works," provides an in-depth look at the architecture and mechanisms enabling real-time collaborative editing in Figma. Below is a structured summary highlighting key concepts and direct quotes from the article:

• Client-Server Architecture: Figma employs a client-server model where web clients communicate with servers via WebSockets. Each document has a dedicated server process managing real-time collaboration.

"We use a client/server architecture where Figma clients are web pages that talk with a cluster of servers over WebSockets." citeturn0search0

• Custom Multiplayer Solution: Instead of using traditional Operational Transforms (OT), Figma developed a bespoke system tailored to design tools, prioritizing simplicity and performance.

"We decided to develop our own solution... we didn’t want to use operational transforms... As a startup we value the ability to ship features quickly, and OTs were unnecessarily complex for our problem space." citeturn0search0

• Document Structure: Each Figma document is organized as a tree, akin to the HTML DOM, with a root object encompassing pages and nested objects.

"Every Figma document is a tree of objects, similar to the HTML DOM. There is a single root object that represents the entire document." citeturn0search0

• Conflict Resolution via CRDTs: Figma's system draws inspiration from Conflict-Free Replicated Data Types (CRDTs), specifically implementing structures like grow-only sets and last-writer-wins registers to manage collaborative edits.

"Figma’s data structure isn't a single CRDT. Instead it's inspired by multiple separate CRDTs and uses them in combination to create the final data structure that represents a Figma document." citeturn0search0

• Property Synchronization: The servers track the latest values for each property on objects, ensuring that non-conflicting edits merge seamlessly, while concurrent edits on the same property resolve using a last-writer-wins approach.

"Figma’s multiplayer servers keep track of the latest value that any client has sent for a given property on a given object." citeturn0search0

• Object Creation and Deletion: Object IDs are generated to be unique across clients, facilitating offline operations. Deleted objects' data is stored in the undo buffer of the client that performed the deletion.

"This system relies on clients being able to generate new object IDs that are guaranteed to be unique." citeturn0search0

• Tree Structure Management: Parent-child relationships are maintained by storing parent links as properties on child objects. Measures are in place to prevent cycles and ensure a valid tree structure.

"The approach we settled on was to represent the parent-child relationship by storing a link to the parent as a property on the child." citeturn0search0

• Fractional Indexing for Ordering: Children's order within a parent is determined using fractional indexing, assigning positions as fractions between 0 and 1, allowing efficient reordering.

"Figma uses a technique called 'fractional indexing' to do this." citeturn0search0

This architecture ensures that Figma delivers a responsive and reliable real-time collaborative design experience.

Introduction and Motivation: Term rewriting with Meander offers an intuitive introduction aimed at everyday software engineers.

> "Meander is heavily inspired by the capabilities of term rewriting languages. But sadly, there aren't many introductions to term rewriting aimed at everyday software engineers."

Basic Concept of Term Rewriting: It transforms data by applying rules that match a left-hand-side pattern to produce a right-hand-side output.

> "The goal of Term Rewriting is to take some bit of data and rewrite it into some other bit of data. We accomplish this by writing rules that tell us for a given piece of data what we should turn it into."

Simple Rewrite Rule Example: A basic rule maps a specific input (e.g., :x) to a designated output (:y).

> "Here is the most simple rewrite rule imaginable. If we are given :x we turn it into :y."

Combining Multiple Rewrite Rules: Multiple rules can be defined to handle various inputs simultaneously.

> "Here we've extended our rewrite to have multiple rules."

Utilizing Variables in Patterns: Variables (prefixed with ?) match any value and allow that matched value to be reused in the output.

> "Here we added the variable ?x to our left-hand-side. Variables start with a ? and match any value."

Pattern Matching on Data Structures: Rules can be crafted to operate on vectors, enabling extraction of specific elements such as the first element.

> "Here we can see some really simple rules that work on vectors of various sizes. We can use this to extract the first element from each."

Rewriting Strategies for Control: Strategies allow precise control over how and when rewrite rules are applied during computation.

> "Strategies let us control how our terms are rewritten."

The Attempt Strategy: The attempt strategy tries to apply a rewrite and, if it fails, returns the original value to handle cases where no match is found.

> "We can fix that by using the attempt strategy. It will try to rewrite and if it fails, just return our value."

Iterative Application with (until =) Strategy: The (until =) strategy repeatedly applies rules until the expression stops changing.

> "What we really want to say is to continue applying our rewrite rules until nothing changes. We can do that by using the (until =) strategy."

Traversal Approaches – Bottom-Up vs. Top-Down: Different strategies, such as bottom-up and top-down, affect the order and frequency of rule application in nested expressions.

> "If we look at the top-down approach, we can see that the top-down strategy actually gets called three times... Our bottom-up strategy however is only called twice."

Tracing the Rewriting Process: The trace strategy provides visibility into each step of the rewriting process for debugging and analysis.

> "We can inspect our strategies at any point by using the trace strategy."

General Computation via Term Rewriting: Term rewriting can express any computable function, exemplified by implementing Fibonacci with Peano numbers.

> "Term Rewriting is a general programming technique. Using it we can compute absolutely anything that is computable."

Code and Execution as Data: It enables code to be treated as data, allowing for introspection of intermediate execution steps.

> "Not only can our 'code' be data more than it can in lisp, but we can actually have our execution as data."

Support for Partial Programs: The paradigm facilitates working with incomplete programs by allowing unimplemented parts to be represented without immediate failure.

> "Term Rewriting also gives us an easy basis for talking about partial programs."

A New Programming Paradigm: Term rewriting represents a uniform, pattern-based approach to programming that challenges traditional distinctions between code and data.

> "Term Rewriting represents a distinct way of programming. It offer s us a uniform way of dealing with data."

You can change permanent notes

What kinds of connections are there?

Permanent notes

bullets vs written prose?

• After annotating a mass of content around a topic (a problem to be solved?)
• I review all the notes and simply group them (using drag & drop? Or canvassing) as I go through them
• Each note goes into a group of similar notes
• These groups might just be the key takes from my research and the outline of my permanent note
• Problems to be solved could be the anchors of my structure
• At any given moment I'm thinking or consuming content in the context of some problem
• Problem could be phrased as goals, challenges or even products to be designed
  • It might be important to choose which way to phrase it or even use aliasing to find it from different state of mind

Alias for permanent notes

Who is?

• Original thinking
• Contains full context and can be understood as a stand-alone unit
• Contains references & sources

For Literature Notes

• Spontaneous thought that occur.
• Only need quick capture
• I'd also add a quick linking/tagging mechanism so it will at least be at the right area

• Bullet point summary in my own words of external content
• I need to read more on how to take notes
• Annotations seems like an effective approach

How to take smart notes

• Collecting material feels more useful than it usually is
  • When I'm collecting references around a topic of interest I feel productive: it is very systematically, efficient, easy to do, and the product is tangible.
  • However it leaves the meaningful work of actually digging into those references for later.
  • Sometimes that later usually doesn't come and if it does:
    • Digging deeply into references isn't cheap so I usually don't dig into most of that references I collected but only to a selected few.
    • This can be looked at as BFS vs DFS strategy.
    • For articles, collection means less than actually reading and writing about the article.
    • For tools, nothing beats actually using the tool.
  • Perhaps my knowledge accumulation strategy should emphasis shortest path to value.

• Comparison of web annotation systems
  • [[Memex]]
    • Desktop
      • UI looks good
      • Live annotations on the site
      • Multiple notes on page
      • AI integration
      • Annotations for videos
    • Mobile
      • Annotations through the app
        • The app seems to keep the integrity of the original content (no content loss)
      • Buggy text input widget
    • Syncing
      • Require Readwise
        • Not bi-directional!
        • Unmodifiable log-based format
  • [[Hypothesis]]
    • Desktop
    • Mobile
      • Bookmarklet is a good option to bypass the need for extension
      • Proxy is also an option
      • Annotations on the original site (no content loss)
      • No annota
    • Syncing
      • Through Readwise
        • Nice format
        • Full control over frontmatter
      • Obsidian plugin
        • No control over frontmatter
  • [[Readwise]]
    • Desktop
      • UI looks old and unmaintained
      • Live annotations on the site
      • Multiple annotation on site can only be achieved by highlights
        • Only single note per page without highlighting
    • Mobile
      • Annotations through the app
        • Some content is lost when importing to the app
      • Text-to-speech
    • Syncing
      • Through Readwise
        • Not bi-directional!
        • Cleaner format then memex

• I too think knowledge should be open for feedback
• Feedback improves quality
• This also increases exposure which creates opportunities for collaboration

• This is true for my KMS as well
• A system of naming and aliasing is very important for discoverablity
• And also prevent duplications

• I'm looking for a Taxonomy of notes and relations
• Andy's Taxonomy can provide some ideas / insight for my Taxonomy

Really?

• Purpose & Motivation

  We show that for a broad class of CRUD (Create, Read, Update, Delete) applications, this gap can be bridged.

  • This work addresses the difficulty many web authors face in creating interactive, data-driven applications due to limited programming and data-schema knowledge.

• Core Contribution

  Mavo extends the declarative syntax of HTML to describe Web applications that manage, store and transform data.

  • The system introduces a novel approach where authors define schemas implicitly by marking up HTML elements, effectively eliminating the need for complex server-side CMS solutions or JavaScript frameworks.

• Related Work Context

  These three systems introduced powerful ideas: extending HTML to mark editable data in arbitrary web pages, spreadsheet-like light computation, a hierarchical data model, and independence from back-end functionality.

  • Mavo differentiates itself from prior efforts (e.g., Dido, Quilt, Gneiss) by combining lightweight computation, purely client-side data management, and seamless integration with arbitrary HTML layouts.

• HTML-Based Declarative Syntax

  We chose to use HTML elements, attributes, and classes instead of new syntax for Mavo functionality because our target authors are already familiar with HTML syntax.

  • Mavo leverages standard HTML attributes—like property, data-multiple, and data-store—to mark elements for data binding, persistence, and editing.

• Editing & Storage

  Mavo can store data locally in the page (data-store=#elementid\), in the browser’s local storage (data-store=\local\), in an uploaded / downloaded file (data-store=\file\) or on one of the supported persistent storage services.

  • Once marked, any element or text becomes directly editable on the rendered page; changes are saved back to the chosen store, enabling WYSIWYG content management within the browser.

• Objects & Collections

  Properties that contain other properties become grouping elements (objects in programming terminology); this permits a user to define multi-level schemas.

  • By adding data-multiple, users enable collection-like behavior, allowing repeated sections or items to be dynamically added or removed.

• Lightweight Computation & Expressions

  Expressions are delimited by square brackets ([]) by default and can be placed anywhere inside the Mavo instance, including in HTML attributes.

  • This spreadsheet-like capability supports referencing other properties, performing aggregations (e.g., sum(), average(), count()), and conditional logic (iff()).

• Implementation Details

  Mavo is implemented as a JavaScript library that integrates into a web page to simulate native support for our syntax.

  • On page load, Mavo constructs an internal tree representation of the HTML and expressions, updating all references reactively whenever data changes.

• User Studies: Structured Tasks

  We found that the majority of users were easily able to mark up the editable portions of their mockups to create applications with complex hierarchical schemas.

  • Participants successfully transformed static HTML pages (for a ‘Decisions’ app and a ‘Foodie log’) into dynamic CRUD applications, achieving a 100% success rate on the basic CRUD tasks.

• Common Difficulties

  Some participants frequently copied and pasted expressions when they needed the same calculation in different places.

  • Most challenges arose around conditionals (iff()) and more advanced expressions (e.g., filtered counts), underscoring that more complex logic remains harder for novices.

• User Studies: Freestyle Tasks

  We asked them to use Mavo to make their mockup fully functional in any way they chose.

  • Users brought their own HTML for an ‘Address Book’ concept. All were able to implement multi-level data (e.g., multiple phone numbers) with minimal changes.

• Direct Manipulation Design

  Instead of crafting a data model and then deciding how to template and edit it, a Mavo author’s manipulation of the visual layout of an application automatically implies the data model.

  • This makes building and editing data-backed pages more natural for users who think first in terms of presentation.

• Intended Audience & Scalability

  Mavo is aimed at a broad population of users. There is no hard limit to what it can do, since its expressions also accept arbitrary JavaScript.

  • The system is best suited to ‘small data’ applications such as personal information management or single-author web publishing, but can also facilitate multi-user scenarios with suitable back-end access controls.

• Semantic Web Alignment

  Mavo syntax for naming elements is based on a simplified version of RDFa… as a result, at runtime any Mavo instance becomes valid RDFa that can be consumed by any program that needs it.

  • Authors who add property attributes gain structured, machine-readable data without explicitly learning RDFa or JSON.

• Planned Improvements

  A more direct way to declaratively express these operations [sorting, searching, filtering] is needed… sorting, searching and filtering were recurring themes.

  • Future work will focus on refining conditional syntax, improving data migrations when HTML structures change, and incorporating advanced multi-user access models.

• Conclusion

  We show that HTML authors can quickly learn use Mavo attributes to transform static mockups to CRUD applications, and, to a large extent, use Mavo expressions to perform dynamic calculations and data transformations.

  • The researchers envision a future where editing and storing structured data via standard HTML becomes ubiquitous, lowering barriers for both novice and skilled web authors.

• Object algebras solve the expression problem in OO languages using simple generics.

  "This paper presents a new solution to the expression problem that works in OO languages with simple generics (including Java or C#)."

• They avoid the need for advanced typing features such as F-bounded quantification, wildcards, or variance annotations.

  "Object algebras use simple, intuitive generic types that work in languages such as Java or C#. They do not need the most advanced and difficult features of generics available in those languages, e.g. F-bounded quantification, wildcards or variance annotations."

• They improve on the Visitor pattern by eliminating accept methods and preserving encapsulation.

  "Object algebras also have much in common with the traditional forms of the Visitor pattern, but without many of its drawbacks: they are extensible, remove the need for accept methods, and do not compromise encapsulation."

• They support retroactive interface implementations, allowing new operations to be added without modifying existing code.

  "By using this simple pattern we can provide retroactive implementations of interfaces to existing code."

• They enable extension in two dimensions: adding new data variants and new operations.

  "There are two ways in which we may want to extend our expressions: adding new variants; or adding new operations."

• They directly implement functional internal visitors, reflecting Church encodings.

  "Object algebras provide a direct implementation of (functional) internal visitors since constructive algebraic signatures correspond exactly to internal visitor interfaces."

• Multi-sorted object algebras support multiple, potentially mutually recursive types and operations as a family.

  "In larger programs, it is often the case that we need multiple (potentially mutually) recursive types and operations evolving as a family."

• Modular combinators, such as union and combine, allow independent extensibility and parallel execution of operations.

  "Sometimes it is useful to compose multiple operations together in such a way that they are executed in parallel to the same input."

• A real-world case study demonstrates their application in a remote batch invocation system integrating RPC, web services, and SQL translation.

  "We have used this technique in implementing a new client model for invoking remote procedure calls (RCP), web services, and database clients (SQL)."

• In conclusion, object algebras offer a lightweight, factory-oriented programming style that scales well while minimizing conceptual overhead.

  "This paper presents a new solution to the expression problem based on object algebras. This solution is interesting because it is extremely lightweight in terms of required language features; has a low conceptual overhead for programmers; and it scales well with respect to other challenges related to the expression problem."

• Denotational Model Overview:

  This paper presents a denotational model of inheritance. The model is based on an intuitive motivation of inheritance as a mechanism for deriving modified versions of recursive definitions.

• Inheritance as Differential Programming:

  Inheritance is a mechanism for differential, or incremental, programming.

• Handling Self-Reference:

  in order for a derivation to have the same conceptual effect as direct modification, self-reference in the original definition must be changed to refer to the modified definition.

• Fixed Point Semantics Foundation:

  Theorem 1 (Fixed Point) If D is a cpo and f ∈ D → D is continuous, then there is a least x ∈ D such that x = f(x).

• Modeling Inheritance via Generators:

  Inheritance is modeled as an operation on generators that yields a new generator.

• Wrapper Mechanism for Modifications:

  The modifications, however, are also defined in terms of the original methods (via super). In addition, the modifications refer to the resulting structure (via self).

• Equivalence of Semantics:

  Theorem 2 send = behave.

• Operational Method Lookup Semantics:

  When a message is sent, the methods in the receiver’s class are searched for one with a matching selector. If none is found, the methods in that class’s superclass are searched next.

• Intuitive Advantages and Broader Implications:

  The primary advantage of the denotational semantics is the intuitive explanation it provides. It suggests that inheritance may be useful for other kinds of recursive structures, like types and functions, in addition to classes.

• Application to Compiler Generation:

  Our denotational semantics of inheritance can be used as a basis for semantics-directed compiler generation for object-oriented languages, as shown by Khoo and Sundaresh (1991).

• Denotational Design as a Real Process

Denotational design is the process that has been elaborately developed by Conal Elliott.

• Core Principle of Stepping Back from Implementation

We don't want to jump in and say, ‘An image is an array of pixels.’ That’s too soon yet that’s where most of us start.

• Abstract Definition of an Image

An image is just a function from a pixel location, so an X, Y coordinate to color, where X, Y are in the real number space.

• Emphasis on Algebraic Properties and Category Theory

He uses algebraic properties and category theory. I think algebraic properties are a very good indicator that you are, ‘on to something’ in the design.

• Incremental, Iterative Refinement

You have to go back and revise and you make an attempt in a certain direction, and you learn something, and you bring that back to the beginning.

• Four Steps of the Denotational Design Process

These are the four steps that I see... This first one is to...like a Zenning out and forgetting all implementation assumptions...Then you explore...Then you align with category theory concepts...Then the final thing is actually implementing it.

• Challenges with Haskell’s Type System

Haskell has no type for real numbers. Most languages don’t...Another thing is, when you’re talking about say, the Monad laws or the Functor laws...there’s no way to do that equality comparison.

• Similar Difficulties in Clojure

I do think it's a little harder than in Haskell, but I also think that most of the design part is happening in your head.

• The Essence of Denotational Design

It’s about going back to first principles, building things up, understanding how things compose, and following a different gradient from what most people use when they design.

• Introduction to Signals & Reactivity

"Signals are value that change over time...The key to a reactive system is that it knows when you set the value; it looks for the set on the specific property, the specific function...and it reads with a function."

• The speaker emphasizes that signals conceptually hold a current value rather than providing a continuous stream of intermediate states.
• Signals update synchronously, ensuring that any consumer remains consistent with the current snapshot of data.

• Immutable vs. Mutable Structures

"We used to learn the framework and then learn the fundamentals kind of thing... in 2014, JavaScript had already taken off and I admittedly didn’t know very much."

• The conversation highlights the contrast between immutable data (where changes create new references) and mutable data (where changes happen in place).
• Immutable updates trigger re-diffing, whereas mutable updates allow granular changes at the specific location.

• Nested Signals & Efficiency

"If we just go in and change Jack’s name to Janet by just setting the name, we only need to re-run the internal effect…it’s only the nearest effect that runs."

• By nesting signals inside signals, individual property changes can avoid re-running the entire component or the entire data structure.
• This nested approach demonstrates how specific effects update only the parts that need to change, improving performance.

• Store Proxies as a “Best of Both Worlds”

"React basically tells you that this is where you end up—when you know that you can cheat it a little bit, you just get past it."

• Using proxies allows for deep mutation with minimal overhead, merging the developer simplicity of an immutable interface with the fine-grained updates of mutable change.
• The speaker points out that many frameworks lack built-in “derived mutable” structures and rely on bridging solutions such as user-space code or specialized stores.

• Map, Filter & Reduce with Signals

"We can also avoid allocations here by only storing the final results... but only if you care about final results."

• Mapping over large datasets illustrates the trade-offs between immutable approaches (always re-map) and mutable approaches (apply partial updates or diffs).
• The speaker notes that “filter” and “reduce” often involve more complete re-runs and may need specialized logic or custom diffs rather than a one-size-fits-all operator.

• Convergent Nodes & Reactive Graphs

"Because signals are a value conceptually, not a stream—...the goal of effects isn’t to serve as a log... it’s a way of doing external synchronization with the current state."

• Computed values (memos) serve as convergence points in the reactive graph. Multiple sources merge into derived data that updates automatically.
• Fine-grained systems need to track only minimal dependencies, but the conversation repeatedly underscores that different transformations (map, filter, reduce) pose unique challenges.

• Async & Suspense Insights

"Run once doesn’t work—there’s no way to avoid scheduling. We need to always throw or force undefined, so we need suspense."

• Lazy asynchronous signals can lead to “waterfalls,” where the second request only starts after the first completes.
• Suspending or temporarily showing old data can prevent blank states but risks double fetching and inconsistent chaining unless carefully guarded.

• Early Returns vs. Control Flow Components

"Early returns... push decisions upwards where further positions are pushed down, so it might not impact React, but it doesn’t lead to a way forward."

• The speaker critiques patterns that rely on returning early in components, arguing they duplicate layout logic and sometimes break optimal reactivity.
• The recommendation is to align with data flow and push condition checks closer to where data is rendered instead of scattering them at multiple return statements.

• Syntax Debates & Framework Convergence

"Syntax in JS frameworks is overrated... we’re at a point now where the looks are so superficial that you can be looking at complete opposites and it looks identical."

• Despite the prevalent notion that “ruin” syntax in various frameworks resembles React, the underlying reactivity mechanics differ significantly.
• Discussion highlights that every major framework—React, Vue, Svelte, Solid—converges on signals or reactivity, yet each approach’s details (mutable vs. immutable, compiler vs. runtime) vary widely.

• Conclusion: Future of Signals & Reactive Systems

"People are starting to wonder if we should just have one super framework now that we mostly agree... but each framework’s identity is in how it approaches these details."

• The speaker underscores ongoing exploration into data diffing, nesting, and push-pull mechanisms to improve performance while simplifying the developer experience.
• Signals and granular reactivity are core to bridging a user-friendly interface with minimal overhead, a goal each framework pursues in its own unique, evolving way.

Summary of "Wrong Way! Choosing a Direction for Datomic Ref Types" by Francis Avila

Problem Statement

• Datomic reference attributes require a direction (e.g., :vehicle/passengers vs. :passenger/vehicle).
• Key question: Which direction should be chosen for optimal performance and usability?

Why Prefer :passenger/vehicle (Lower-Cardinality Forward Direction)?

1. Keeps collections in map-projections smaller

• "Lower-cardinality attributes values are generally easier to deal with because entity-walking with d/entity or d/touch won’t occasionally give you unexpectedly large sets."
• Key Issue: High-cardinality attributes can create large collections, making entity exploration cumbersome.
• Protection Mechanism: "d/pull protects you from this because it will only pull 1000 items by default, but this is easy to forget!"

2. Keeps EAVT index smaller per entity

• "A high-cardinality relationship between two entities often implies some kind of containment relationship."
• Key Benefit: If the container entity is "rich" (i.e., has many attributes), using :passenger/vehicle keeps the EAVT index for the container smaller.
• Why it matters: Large EAVT indexes decrease index selectivity, making entity lookup less efficient.
• Caveat: This mainly affects d/entity and d/pull, not d/q or d/pull-many, which rely more on AEVT.

3. Improves EAVT history readability

• "Keeping the EAVT smaller per E also makes the history of an entity more human-legible when using d/history database reads over the EAVT index."
• Problem with :vehicle/passengers: "It will clutter the history of the container entity (the vehicle)."
• Advantage of :passenger/vehicle: Changes in passenger-vehicle relationships are recorded on the passenger’s history, not the vehicle’s.
• Takeaway: For audit logs and admin views, the passenger's history is usually more useful than the vehicle's.

4. Supports cardinality-one constraints with last-write-wins semantics

• "Very often, there is also an 'only in one container' constraint between a container and contained entity."
• Key Advantage: Using :passenger/vehicle as a cardinality-one attribute naturally enforces the rule that a passenger can only be in one vehicle at a time.
• Contrast with :vehicle/passengers: This direction cannot enforce uniqueness without additional constraints (e.g., transaction functions).
• Takeaway: If the relationship is inherently one-to-one, :passenger/vehicle provides better integrity guarantees.

Why Prefer :vehicle/passengers (High-Cardinality Forward Direction)?

1. Better schema legibility

• ":vehicle/passengers makes it clear when grouping by keyword namespace that vehicles are expected to reference many passengers."
• Issue with :passenger/vehicle: It does not make the vehicle-passenger relationship explicit.
• Challenge: No built-in way to highlight reverse ref relationships in Datomic schema.
• Possible Fixes:
  • Use entity specs with :doc metadata.
  • Create custom ref-range annotations.

2. More useful d/index-pull queries

• "d/index-pull provides extremely efficient, lazy, and offset-able pulls over the third slot in an AVET or AEVT index span."
• Problem: "d/index-pull cannot scan VAET."
• Why it matters: If you need to retrieve all passengers for a vehicle, d/index-pull won’t work efficiently for :passenger/vehicle.
• Workaround: Adding :db/index true to :passenger/vehicle, but this adds extra datoms to the index.

3. Reduces index segment churn

• "If the number of containers is significantly smaller than the number of contain-able entities, and the containment relationship churns frequently, the lower-cardinality-forward attribute is going to invalidate more segments during indexing."
• Example: If passengers frequently switch vehicles, using :passenger/vehicle means frequent updates across many EAVT and AEVT segments, causing more index fragmentation.
• Alternative: ":vehicle/passengers results in fewer index segments being invalidated."
• Takeaway: For high-churn relationships, :vehicle/passengers may be more efficient.

Summary Takeaway

• :container/contained (e.g., :vehicle/passengers) is intuitive, but :contained/container (e.g., :passenger/vehicle) is usually better due to:
  • Smaller map-projections.
  • More efficient entity walking.
  • Better historical auditing.
  • Natural enforcement of cardinality constraints.
• But :vehicle/passengers is better when:
  • Schema readability is critical.
  • d/index-pull is needed for efficient queries.
  • The relationship has frequent changes and involves many contained entities.

• Introduction to the Stream and Purpose

“I'm uh pretty excited about this one because I'm going to get to finally show off some of the stuff I've been working on for months.”

• Highlights the speaker’s enthusiasm to demonstrate months of development progress.
• Establishes that the stream will cover new reactive features and mechanisms.

• Parallel and Nested Async Fetching

“What if we want to do nested fetching where each component fetches data, but we don’t want to cause waterfalls? … We’ve basically solved waterfalls because promises do not throw out too early.”

• Emphasizes that asynchronous tasks in Solid can now run in parallel.
• Shows how nested components fetch data without blocking each other.

• createAsync as a Core Signal Primitive

“createAsync… if we look at the signature here, it expects a computation… and then returns an accessor of a number… it will just give you the resolved async value without returning undefined.”

• Introduces createAsync as an “async signal” that never yields undefined.
• Allows a direct, non-nullable way to fetch and use async data in the component.

• Local vs. Global Suspense Boundaries

“We don’t have to throw away our render tree just because we have something async… it only throws exactly where we read it, and that means everything else is fine.”

• Suspense is granular: only the part that reads an unresolved value suspends.
• Other parts of the UI remain interactive rather than unmounting the entire tree.

• Self-Healing Error Boundaries

“We basically collect the nodes that fail, and then, if they become unfailed or get disposed, the boundary can remove itself—self-heal.”

• Explains that failed async or errors get tracked locally by boundaries.
• Once the failure resolves or is disposed, the error boundary automatically resets.

• Avoiding Unpredictable Tearing

“You basically never want your async data to just flicker in or out. We can choose to throw or to keep ‘stale’ data. Suspense can opt into that.”

• Details the importance of consistent state during asynchronous updates.
• Introduces a mechanism (isStale or latest) to avoid jarring UI replacements.

• Splitting createEffect for Predictability

“If we just let you read signals in the same function where we do side effects, we get unpredictable re-runs… so we split it into two halves.”

• Shows how Solid 2.0 separates the tracking (pure) side from the side-effect (impure) side.
• Ensures that data retrieval and side-effects remain consistent, avoiding “zalgo” outcomes.

• Mutable Reactivity and Store Projections

“I realized a store approach is a general solution… the idea is you have a single source signal and can ‘project’ it out to many places… only the fields that change update.”

• Describes a new technique called “projections” to handle large data sets efficiently.
• Allows per-field reactivity, so only the row or property that changes triggers updates.

• Granular Handling of Async and Errors

“Error boundaries and suspense handle each failing effect locally. The rest of the system doesn’t even know something failed.”

• Illustrates that errors remain localized, preventing a full unmount.
• Reflects the fine-grained reactivity approach, making error handling more targeted.

• Impact on Ecosystem Comparisons

“React can’t do this because… they don’t have the semantics to pull from signals. It’s not the same model.”

• States the fundamental difference from React’s component rendering.
• Emphasizes that granular updates and specialized async signals differ sharply from React’s design.

• Future Plans: SSR, Hydration, Transitions

“We still need transitions. I haven’t implemented them yet, but they’re part of the equation. … Also looking at SSR so we can skip hydration IDs.”

• Points to upcoming work for Solid 2.0: concurrency transitions, improved server rendering, and more efficient hydration.
• Aims to unify the new runtime mechanisms with advanced features like streaming.

• Concluding Observations

“We’ve basically… proven we can handle async, error boundaries, and concurrency all purely at the reactive level. This changes everything.”

• Summarizes the significance of these new developments in Solid’s reactivity engine.
• Stresses that purely runtime-based solutions enable advanced use-cases without a compiler-centric approach.

Summary of the Talk: Building More Powerful User Interfaces in the Browser

Introduction & Motivation

• The speaker reflects on a year’s work in adopting modern web technologies (AMD, Backbone, HTML5, CSS3, new browser APIs) but realizes that the fundamental power given to users has not improved significantly from 15 years ago.

"I looked at what we had built at the end of the year and I said you know I think I could have built this 15 years ago when I started writing JavaScript."

• Traditional web applications function like simple forms: users provide inputs, and the application computes outputs. The speaker seeks a way to eliminate this rigid distinction and create more interactive and dynamic UI models.

"You have to decide in advance which things you think are input... and which things you think are output."

Example: Federal Budget Visualization

• A web-based visualization of the US 2013 federal budget illustrates the limitations of traditional input-output models.

"What if we could actually take this and change this and not have a distinction between input and output?"

• The speaker explores how users could interact more dynamically by locking certain variables (e.g., keeping the deficit fixed) and observing how other variables (e.g., taxes) adjust automatically.

Concept of Constraint Programming

• Constraint programming allows defining relationships between variables instead of prescribing explicit procedures for computation.

"Constraint programming is about writing our programs in terms of relations instead of procedures."

• Example: Instead of coding tax rates explicitly, define relationships between tax brackets and let the system adjust them dynamically.

"We make all of our variables both input and output."

Cassowary Solver for UI Constraints

• Cassowary (CJS), a JavaScript library, enables constraint solving for UI applications. Originally used in iOS Auto Layout.

"Cassowary is a fantastic library for doing constraint programming in JavaScript."

• It ensures relationships like total spending = defense + non-defense spending remain consistent, even when one variable is modified dynamically.

"The solver will automatically for us make sure that this relationship between the three variables always holds."

• Supports priority constraints to handle over-constrained systems (e.g., some constraints are "required," others are "nice to have").

"If you have problems that are over constrained where there's no complete solution, Cassowary will find you the best solution you can find."

Practical Implementation

• Basic interaction model:
• Mark a variable as being edited (beginEdit).
• Suggest new values (suggestValue).
• End editing (endEdit).
• Add constraints (stayConstraint) to keep certain variables fixed.

"The reason this is called Suggest and not set... is that this value might not lead to an actual solution."

• Using constraints simplifies complex relationships, such as progressive taxation and revenue calculations.

"Several very natural things fell out of writing the relationships that would have been a real pain to code by hand."

Limitations of Cassowary

• Only supports linear equations and numeric variables (e.g., it cannot handle quadratic constraints).

"C can only solve problems where the variables are numbers and it can only solve where the relationships between the numbers are linear expressions."

• Proposes improvements:
• Nonlinear constraint solvers (for geometric problems, e.g., keeping a point on a circle).
• MiniKanren & Core.logic for relational programming on non-numeric problems.

MiniKanren & Core.logic

• A relational programming model that generalizes constraints beyond numbers to trees, lists, colors, and abstract structures.

"MiniKanren provides relational programming just like we've been doing with Cassowary but over non-numerical problem domains."

• Example: Sudoku Solver in Core.logic solves the problem declaratively by defining constraints rather than writing procedural code.

"This looks like a statement of the problem of Sudoku, and yet this will run really fast and give us the answers."

• Benchmarks: A JavaScript Core.logic Sudoku solver runs 100x faster than Peter Norvig’s optimized Python version.

"This version which is not handwritten just happens to use a constraint solver works on average about a hundred times faster than Peter Norvig's Python code in JavaScript in the browser."

Future Directions: Cooperating Solvers

• Alan Kay’s Viewpoints Research Institute explores "cooperating solvers," where different constraint-solving techniques (dataflow, logical constraints) work together.

"How do they cooperate? There's some really interesting PDFs and example code showing how to do that."

• Potential applications:
• More powerful layout engines.
• Interactive problem-solving beyond algebraic constraints.

Bonus: Brett Victor’s Scrubbing Calculator

• Inspired by Brett Victor’s UI work, the speaker demonstrates an interactive "scrubbing calculator" implemented in Cassowary.

"This is an example of the kind of program that would be hard to write by hand unless you really want to write your own computer algebra system."

• Allows solving for any variable in an equation dynamically just by adjusting values interactively.

"If we had a real computer algebra system in JavaScript wouldn't that be great?"

Conclusion

• Advocates for constraint programming as a means to build more powerful, flexible UI applications.
• Suggests integrating constraint-based approaches in mainstream web development to create more intelligent, adaptive, and user-driven applications.

"If we want to build more powerful applications, we’ve got to give our users more leverage."

• Calls for open-source contributions to Cassowary and similar projects.

"C.JS is a fantastic one... let's go out and build more powerful user interfaces."

Summary of DevTools FM Podcast with Juan Capa on Membrane.io

Introduction and Background

• Juan Capa is the creator of Membrane.io, a still-in-development platform for simplifying API automation and internal tooling.

"Juan is the creator of membrane.io, a still-on-development platform for simplifying API Automation and internal tooling."_