The performance of the system must be reliable and controllable. Its behavior should be safe, and the way it is designed and used should be ethical [768]. Users need to trust the system's decisions and ability. It should be made clear to the user what it can and cannot do.

such systems should be designed to take into account the fact that automated results will inevitably be incorrect on occasion.

users can be trained to understand not only the decision-making tasks but also the underpinning capabilities and limitations of the automation solution.

automated systems that indicate when automation may fail or has failed are more likely to gain an appropriate level of trust from users.

When the system fails, users need to be able to redirect it. To avoid biases and discrimination, some level of transparency and explainability is required.

For small entrepreneurs in the US, deciding what to sell and where to make it has traditionally been a slow, labor-intensive process that can take months. Now that work is increasingly being done by AI tools like Accio, which help connect businesses with manufacturers in countries including China and India.

Zhang, of Alibaba.com, says Accio currently does not include advertising. Suppliers can pay for higher placement in Alibaba.com's regular search results, but Zhang says Accio is 'not integrated' with that system.

Sellers say that while AI tools have made it easier to come up with ideas and get a business off the ground, they do not replace the core skills that make someone good at e-commerce.

Sally Li, a representative at a makeup packaging company in Wuhan, China, says her firm has started writing more detailed product descriptions and adding information about its equipment and manufacturing experience on Alibaba.com because it suspects those details make its listings more likely to be surfaced by AI.

McClary took the process from there, contacting the supplier himself to discuss the revised design. Within a month, the new version of the Guardian flashlight was back up for sale on Amazon and on his brand's website.

current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction

amplifies the false narrative that technology and creativity are at odds, and that existing rights holders must be compensated by AI companies for changing industry dynamics.

The government has so far favoured a pro-innovation, sector-led approach, prioritising voluntary principles over hard regulation.

introducing a commercial text and data mining exception for AI training would expand the AI sector in the country.

The government has so far favoured a pro-innovation, sector-led approach, prioritising voluntary principles over hard regulation.

This article argues that squirrel ecology offers a sharp comparative case because arboreal locomotion, scatter-hoarding, and audience-sensitive caching couple all three demands in one organism.

Agentic AI is increasingly judged not by fluent output alone but by whether it can act, remember, and verify under partial observability, delay, and strategic observation.

We introduce a minimal hierarchical partially observed control model with latent dynamics, structured episodic memory, observer-belief state, option-level actions, and delayed verifier signals.

Existing research often studies these demands separately: robotics emphasizes control, retrieval systems emphasize memory, and alignment or assurance work emphasizes checking and oversight.

Agentic AI is increasingly judged not by fluent output alone but by whether it can act, remember, and verify under partial observability, delay, and strategic observation.

We've seen customers go from 10-20% field accuracy with a frontier model to 99-100% just by switching to using Reducto's Deep Extract.

The issue isn't that models are bad at reading documents. It's that single-pass extraction has no mechanism to catch its own mistakes, and models get lazy.

For the documents that matter most, it gets to 99–100% field accuracy, even out-performing expert human labelers on extraction tasks.

The demand for these medications has been the most ferocious thing I have witnessed in my working life, and the hardest parts of running a telehealth company, like finding doctors and fulfilling prescriptions, can be entirely outsourced to platforms like CareValidate and OpenLoop.

His affiliates, armed with AI, built fake doctor profiles in Meta ads and made unscrupulous claims about weight loss using fake testimonials.

The cost of understanding what happens in a video has dropped by a factor of roughly 40, while the quality of that understanding has improved dramatically.

Historically, AI evaluation has leaned toward the forest approach. Most researchers settle for 1 to 5 raters per item, assuming this is enough to find a single 'correct' truth.

Reconstructing raw inputs forces models to model irrelevant low-level detail. Predicting in a learned embedding space allows the model to focus on semantically meaningful, causally relevant features.

Whether or not this specific bet pays off, the underlying argument that the next meaningful leap in AI capability requires moving beyond language modeling is increasingly hard to dismiss.

AMI Labs is not building a product for immediate deployment. This is a fundamental research effort, likely measured in years before commercial applications emerge.

LLMs have no grounded understanding of the physical world. They model the statistical distribution of language about reality, not reality itself.

Whether or not this specific bet pays off, the underlying argument that the next meaningful leap in AI capability requires moving beyond language modeling is increasingly hard to dismiss.

The clustering of capital and talent around this problem is itself a signal. The applications that most clearly benefit from world models are those where LLMs have struggled most.

AMI Labs is not building a product for immediate deployment. This is a fundamental research effort, likely measured in years before commercial applications emerge.

LLMs have no grounded understanding of the physical world. They model the statistical distribution of language about reality, not reality itself.

You have to have people that have the ability to rethink the workflow at a scale that AI can execute, versus at a scale that humans can execute.

95% of organizations are getting zero return on AI deployed, with most failures found due to 'brittle workflows.'

in 2024, 47% of AI solutions were built internally and 53% were purchased; today, 76% of all AI is purchased rather than developed in-house.

You have to have people that have the ability to rethink the workflow at a scale that AI can execute, versus at a scale that humans can execute.

95% of organizations are getting zero return on AI deployed, with most failures found due to 'brittle workflows.'

in 2024, 47% of AI solutions were built internally and 53% were purchased; today, 76% of all AI is purchased rather than developed in-house.

You don't need a separate agent API. You need to look at every `input()` call, every CWD assumption, every pretty-printed-only output, and ask: what if the user on the other end is a process, not a person?

Implicit state is the Enemy

The funny part is that none of this made the CLI worse for humans. The TUI picker still works and looks fancy, progress spinners still spin, confirmation dialogs still confirm. We just added a second door.

Every prompt is a flag in disguise

Designing for agents forced us to build better tools for everyone.

The funny part is that none of this made the CLI worse for humans.

Designing for agents forced us to build better tools for everyone.

Consequently, they cannot verify if tools were actually invoked, applied correctly, or used efficiently.

Experimental results show the best model, Gemini3-pro, achieves 56.3% overall accuracy, which falls significantly to 23.0% on Level-3 tasks

However, existing evaluations fall short: they lack flexible tool integration, test visual and search tools separately, and evaluate primarily by final answers.

the inherent limitations of such a single-paradigm approach pose a fundamental challenge for existing models

these two challenges are fundamentally distinct: the former relies on fuzzy semantic planning, while the latter demands strict logical constraints

existing methods typically attempt to address both issues simultaneously using a single paradigm

computer-use agents extend language models from text generation to persistent action over tools, files, and execution environments

current systems remain highly vulnerable

intermediate actions that appear locally acceptable but collectively lead to unauthorized actions

model alignment alone does not reliably guarantee the safety of autonomous agents.

computer-use agents extend language models from text generation to persistent action over tools, files, and execution environments

current systems remain highly vulnerable

intermediate actions that appear locally acceptable but collectively lead to unauthorized actions

harmful behavior may emerge through sequences of individually plausible steps

model alignment alone does not reliably guarantee the safety of autonomous agents

让你能像导演一样控制 AI 视频的每个环节

AI Agent 可以通过标准 MCP 协议直接读取和操作 𝕏 平台：搜索推文、发帖、查看用户信息、管理书签、收发私信等。

内置视频和音乐生成

记忆系统学会了"做梦"

内置视频和音乐生成 记忆系统学会了"做梦"

The AI is actually very good at this, especially if you have a conversation with it beforehand. That's what Ask mode is for.

Sandboxes made for running tens of thousands of agents

谷歌在沉寂了很长时间以后，终于发了一个不错的模型，而且还是开源的 Gamma 4 系列。专门用来在本地设备（比如手机、电脑）上跑

Claude 的 Max Pro 账号额度不允许给第三方产品用了，如果你没有使用 Agent SDK 和 Claude Code 为底座的产品，就不能用这个账号里的额度

I feel confident, though, that the slippery feeling people associate with AI products is a solvable problem, and the solution looks more like thoughtful interface design than better models. The models will keep improving on their own. The harder work is building the structure around them so that their output feels reliable, legible, and trustworthy.

When you delegate an issue to an agent in Linear, the delegation is visible. There's a person who set the agent loose within that system, and that person is accountable for the outcome. You design the environment well, you let the agent run, and you own what it produces.

The more important work happens before the agent even starts. An agent operating inside a well-designed system already has the context and constraints it needs to do good work. In Linear, that means project plans, issue backlogs, code, and documentation. These all shape what the agent does and how it does it.

An agent cannot be held accountable. I think about this principle most. The instinct to put a human in the loop is understandable, but taken literally, it can mean a person approving every step before anything moves forward. The human becomes a bottleneck, rubber-stamping work rather than directing it, and you lose much of what makes agents valuable in the first place.

The first interface that spread for AI tools was the chat window. That makes sense. When you don't know what something can do, the safest approach is to let people ask. A conversation feels familiar, it stretches across many situations, and it doesn't force a specific structure up front.

Non-deterministic software breaks the contract. When outcomes can vary, sometimes wildly, based on what someone types into the same chat window, designing for reliability becomes genuinely harder. This slippery feeling is the design problem of this era, and it almost always traces back to the interface rather than the language model—which means it belongs to designers, not researchers.

AI is a way to level the playing field, for sure! Successful writers have always operated with a lot of support around them, but not everyone has access to those resources.

When I sit down to write a piece, and before I even write a word, I have the agent interview me. It asks questions to draw out what I'm thinking about the topic.

It has a panel of critics who tear my work apart from different angles—skills I wrote to invoke certain kinds of feedback, whether it's for length, pacing, or the soundness of the argument.

My process has about as much in common with that as cooking has with microwaving a frozen dinner.

Research is thinking. Outlining is thinking. Writing is thinking. Any portion of that done by AI is less thinking done by you.

both companies are hinting that these models are a real step forward, not just small upgrades.

Gemma points in the opposite direction: smaller models, local compute, more ownership.

A founder in LA reportedly scaled Medvi toward $1.8B in annual sales with basically one full-time employee.

And once models get good at that, the question stops being whether they can make beautiful images. It becomes whether people still notice when something was never real to begin with.

The first wave of image models was mostly about making cool-looking images. This next phase is about making ordinary things look real.

We are building a world where machines write the code, machines choose the dependencies, and machines ship the updates. The AI agents are building the software. If we don't secure the supply chain they rely on, the AI agents are cooked.

The autonomous coding agents now entering production can install dependencies, execute builds, and open pull requests without a human ever touching the keyboard. They optimize for 'does this work?' not 'is this safe?'

Hallucinated packages are the sleeper threat. LLMs regularly invent package names that don't exist. One study found that nearly 20% of AI-recommended packages were fabrications, and 43% of those hallucinated names appeared consistently across queries.

AI agents select known-vulnerable dependency versions 50% more often than humans. Worse, the vulnerable versions they pick are harder to fix, requiring major-version upgrades far more frequently.

Talent density : the biggest prizes in capitalism attract the best minds in the field. These are the fastest growing software companies in history.

At this rate, the phone in your pocket will run today's frontier models before you upgrade it.

In 23 months, the same capability that needed 1.8 trillion parameters now fits in 4 billion parameters. A 450x compression.

Within three to four months, you can run a model with similar performance on your laptop; 23 months later, you can run the same model on your phone.

a free model that matches GPT-4o and runs entirely on your phone

Exposure alone is a completely meaningless tool for predicting displacement

in the past year Huawei has overtaken Nvidia as the leading source of AI computing power in China, at least in terms of rated FLOP/s

We estimate that as of the end of 2025, Chinese companies collectively own just over 5% of the cumulative computing power of the leading AI chips sold in recent years

Many frontier AI developers, including Anthropic and OpenAI, acquire almost all of their compute from hyperscalers and other cloud providers.

We estimate that over 60% of global AI compute (in terms of total computing power) is owned by the five US hyperscalers, led by Google.

複雑なリサーチは、単一のクエリに対する回答の集積ではなく、アイデアの生成から、裏付けとなる証拠の探索、矛盾の解消、そして最終的なレポートとしての構造化まで、一連のプロセスを完遂する必要があります。

推論時により長く、深く思考させることでよりよいアウトプットを引き出せる。これが推論スケーリングの本質です。

For higher-interactivity scenarios, execution time for MoE models is bound by expert weight load time. By splitting, or sharding, the experts across multiple GPUs across NVL72 nodes, this bottleneck is reduced, improving end-to-end performance.

NVIDIA was the first and only platform to submit DeepSeek-R1 results on MLPerf Inference when the benchmark debuted last year.

The E4B and E2B are the newest edition of on-device and mobile designed models first launched with Gemma 3n.

Modern physical AI agents are evolving rapidly with Gemma 4 models that integrate audio, multimodal perception, and deep reasoning capabilities.

By using SAM, the Alta team has been able to process more than 20 million images without incurring exorbitant costs, allowing them to focus on building the best possible product for their users.

If we knew that every image uploaded was a beautiful model shot, segmentation would be far easier, but because of the nature of user-uploaded content, we need the best possible segmentation.

The edge models feature a 128K context window, while the larger models offer up to 256K

Within ChatGPT Business and Enterprise, the number of Codex users has grown 6x since January.

Codex-only seats have no rate limits, and usage is billed on token consumption.

Priority areas include safety evaluation, ethics, robustness, scalable mitigations, privacy-preserving safety methods, agentic oversight, and high-severity misuse domains.

Fellows will receive API credits and other resources as appropriate, but will not have internal system access.

Fellows will work closely with OpenAI mentors and engage with a cohort of peers.

We are especially interested in work that is empirically grounded, technically strong, and relevant to the broader research community.

Demand from Claude customers has accelerated in 2026. Our run-rate revenue has now surpassed $30 billion—up from approximately $9 billion at the end of 2025.

The real monster

There's a fundamental problem with these tools beyond the capacity of any deployment strategy to solve: the tool requires expertise to validate, but its use diminishes expertise and stunts its growth

The thing about agentic coding is that agents grind problems into dust. Give an agent a problem and a while loop and - long term - it’ll solve that problem even if it means burning a trillion tokens and re-writing down to the silicon. Like, where’s the bottom? Why not take a plain English spec and grind in out in pure assembly every time? It would run quicker. But we want AI agents to solve coding problems quickly and in a way that is maintainable and adaptive and composable (benefiting from improvements elsewhere), and where every addition makes the whole stack better. So at the bottom is really great libraries that encapsulate hard problems, with great interfaces that make the “right” way the easy way for developers building apps with them. Architecture! While I’m vibing (I call it vibing now, not coding and not vibe coding) while I’m vibing, I am looking at lines of code less than ever before, and thinking about architecture more than ever before. I am sweating developer experience even though human developers are unlikely to ever be my audience. How do we make libraries that agents love?

Anthropic, the company behind the Claude AI model that was integrated into Palantir’s Maven Smart System, published a landmark paper on the problem in 2023. “Towards Understanding Sycophancy in Language Models,” presented at ICLR 2024, demonstrated that five state-of-the-art AI assistants consistently exhibited sycophantic behaviour across four varied text-generation tasks. The researchers found that when a response matched a user’s pre-existing views, it was significantly more likely to be rated as “preferred” by both humans and the preference models used to train the AI. Both humans and preference models, the paper concluded, prefer convincingly-written sycophantic responses over correct ones “a non-negligible fraction of the time.

A growing body of evidence, drawn from leaked planning documents, academic research, and the testimony of intelligence professionals, suggests that the most consequential military operation of the twenty-first century may have been shaped less by strategic necessity than by a phenomenon researchers now call AI sycophancy — the tendency of large language models to tell their users exactly what they want to hear.

Our preliminary results indicate that there is an additional phase, the intention to learn, and three relating factors, self-efficacy, conversion readiness, and peer support, that significantly influence the acceptance of mobile technologies among the participants, but are not represented in the existing models. With these findings, we propose a tentative theoretical model that extends the existing theories to explain the ways in which our participants came to accept mobile technologies.

Triangulating the empirical findings from our preliminary results with the existing theoretical models, we proposed an extension of the existing theoretical models that explains the technology acceptance behavior of our participants who were aged 60 or over. Our proposed model incorporates key elements of prior models and introduces novel components that significantly influence the participants' technology acceptance, namely one new phase, intention to learn, and three factors, self-efficacy, conversion readiness and peer support.

Consolidating our preliminary findings with the existing models, we propose an extended technology acceptance model for older adults illustrated in Figure 3. Extending to the predecessor theories, our tentative model introduces the perceived effort of learning a new technology as an obstacle for older adults' technology acceptance, which has not been reported in any studies of younger adults' technology acceptance.

Another stream of efforts sought to understand physical and cognitive performance of older adults in interacting with mobile technologies. Studies have shown that typical interaction components and techniques of a smartphone often prevent older adults from smooth and instant interactions with it. For example, the small size and the low contrast of buttons on a mobile display has a significant negative influence on interaction performance such as speed and accuracy [18], and decline in motor skills is correlated with time required to complete a task [30].

Lee and Coughlin reviewed studies of older adults' technology acceptance and identified ten factors that are critical facilitators or determinants of older adults' acceptance of technology: value, usability, affordability, accessibility, technical support, social support, emotion, independence, experience, and confidence [20].

Many studies have empirically investigated technology acceptance practices among older adults. While diverse in detail, most works point out that an individual's personal context [38] and the social context [36] in which the technology is introduced are the primary factors influencing the perception of, experience with, and evaluation of new technological developments among older adults [19].

Seniors have historically been late adopters to the world of technology compared to their younger counterparts [24, 40]. As a result, older adults and their adoption of new technologies have been a topic of active research since the advent of consumer technologies (e.g., automated teller machine [32], scanner-equipped grocery stores [41], electronic funds transfer [15]).

Nowadays, older adults are increasingly adopting and adapting to information and communication technologies [5]. For example, smartphone ownership among older adults has significantly risen in recent years [3]. However, its adoption levels among older adults in the US still sit at 27% as of 2015, whereas some 85% of Americans aged 18-29 are smartphone owners [31].

Consolidating our preliminary findings with the existing models, we propose an extended technology acceptance model for older adults illustrated in Figure 3. Extending to the predecessor theories, our tentative model introduces the perceived effort of learning a new technology as an obstacle for older adults' technology acceptance, which has not been reported in any studies of younger adults' technology acceptance.

our key focus is to build a theoretical model that explains the process through which older adults accept (or reject) mobile technology, which can provide theoretical guidelines when designing a technology, and which may also be able to generate new investigations and experiments.

Triangulating the empirical findings from our preliminary results with the existing theoretical models, we proposed an extension of the existing theoretical models that explains the technology acceptance behavior of our participants who were aged 60 or over.

Again following grounded theory practices from [33], we compared the model that emerged from our data with existing theoretical models of technology acceptance to determine differences and similarities between them.

Employing the grounded theory method [33], we allowed recurring themes and concepts in relation to technology acceptance behaviors to arise from the data itself. Then, by triangulating our empirical findings with existing theoretical models from the literature, we found out that the existing models of technology adoption require new theory components to be able to describe technology adoption processes of our participants.

Using TAM, UTAUT, and several other works as theoretical underpinning, Renaud and Biljon proposed a model to explain older adults' mobile phone adoption.

Extending the original TAM and consolidating the constructs of several other existing models, Venkatesh et al. proposed the Unified Theory of Acceptance and Use of Technology (UTAUT) [37].

We propose that cognitive engagement may be a useful construct in conceptualizing human engagement with AI and can help to distinguish between passive engagement, when individuals simply follow AI recommendations, and deeper forms of engagement, when they critically examine these recommendations and compare them with their own knowledge and judgement.

An outcome of deeper cognitive engagement would be an ability to reject information that is inconsistent with individuals' own knowledge and beliefs, and to adjust their own knowledge to incorporate new information.

Given continuous concerns regarding the reliability and trustworthiness of AI, human critical engagement may be a necessary component of successful human-AI interaction, particularly in domains with a high cost of errors, such as health and medicine.

In many areas of human enterprise, individuals increasingly rely on Artificial Intelligence (AI) to inform their decisions and choices.

How do people process the information and advice they receive from AI, and do they engage with it deeply enough to enable learning?

When people receive advice while making difficult decisions, they often make better decisions in the moment and also increase their knowledge in the process.

Incidental learning typically occurs as a byproduct of other activities (e.g., problem solving, advice seeking) rather than as a result of explicit or formal educational activities [47]. However, like formal learning, incidental learning can only occur if people engage deeply with information.

This would suggest that this design did not fully reach the constructive level from the ICAP framework [15, 16].

While prior work has highlighted the critical role of explanations in promoting learning [10, 18], our work additionally demonstrated the value of creating the conditions for learners to engage constructively (as defined in the ICAP framework [15, 16]) with the explanations.

We hypothesize that the observed difference in learning gain was due to the degree of cognitive engagement with AI-generated information. When individuals were provided with a solution to their task (in the form of a decision recommendation), they did not need to engage deeply with the explanations and could simply proceed with action. However, when they needed to arrive at their own decisions, they needed to engage with the provided explanations and synthesize the information to arrive at the conclusions.

Rotgans and Schmidt attributes these differences in cognitive engagement to different degrees of autonomy afforded by different learning tasks [59].

While some authors discuss cognitive engagement as a personal trait of a student that does not depend on context [3], others suggest that cognitive engagement depends on the structure of each task [15, 30, 59].

al. propose Interactive-Constructive-Active-Passive (ICAP) framework to describe a continuum of learning behaviors (from passive, to active, to constructive, to interactive) and argue that each subsequent level leads to an increase in cognitive engagement and learning [15, 16].

Research in cognitive psychology suggested that people process information on different levels. Deep processing occurs when individuals engage in more meaningful analysis of information and link it to existing knowledge structures [2]. In learning sciences, depth of processing is often associated with the degree of cognitive engagement, which is described as a "psychological state in which students put in a lot of effort to truly understand a topic and in which students persist studying over a long period of time." [59].

Researchers in learning sciences use the term "cognitive engagement" to describe learners' engagement with the learning process. When people are cognitively engaged with instructional process and materials, they are more likely to benefit from instruction and are more likely to acquire new skills and knowledge.

Addressing this feedback, by training more abstractive CTR models or performing a post-hoc abstraction of the generated summary, is an interesting future direction we plan to explore.

Additionally, our tool currently helps users in the reviewing step solely with the alignment functionality. Future work should add additional assistance during this step in the form of suggested improvements to selected unsatisfactory content in the summary, in addition to the alignment feature.

Future work should expand the application's capabilities to the multi-document setting, both in terms of the backend models and in terms of accessibility and intuitiveness of the application's frontend design.

Lastly, exploring strategies to scale SUMMHELPER to a multi-document setting presents another promising avenue for future investigation.

Additionally, in light of some user feedback, another interesting extension includes developing more abstractive consolidation and fusion models, which would offer control over the level of abstractness in the outputs.

Future work may include investigating more effective semantic strategies to locate summary-source alignments with acceptable latency.

highlights are incorporated into the input text with special markups, <extra_id_1> and <extra_id_2>, marking the beginning and end of each highlighted span, respectively. In our configuration, we set the maximum input length to 4096 and the maximum target length to 400. A greedy decoding strategy was used in order to optimize the decoding speed.

Our approach locates the longest common subsequence (LCS) between the lemmas of each input sentence and each summary sentence, followed by several heuristics to filter out irrelevant LCSs

For the summarization model, we used a BARTlarge model (Lewis et al., 2019) trained on the CNN/Daily Mail dataset (Hermann et al., 2015), selected for its noticeable popularity.

For the initial auto-consolidation, we deploy an available Controlled Text Reduction model (Slobodkin et al., 2023), which is a Flan-T5large model (Chung et al., 2022), finetuned on the highlights-focused CTR dataset.

we deploy the ExtractiveSummarizer model from the TransformerSum library. The model, a RoBERTabase (Liu et al., 2019) trained on the CNN/DailyMail summarization dataset (Hermann et al., 2015), operates as a binary classifier.

This step coincides with the recently introduced Controlled Text Reduction task (CTR; Slobodkin et al., 2022), which produces a coherent fused version of the content of marked spans ("highlights") in a source document, as interpreted within the context of the full text.

SUMMHELPER is a modular system consisting of separate components, each performing one subtask, allowing user modifications of that sub-task's output. Such decomposition has been studied before in the context of fully automated summarization, with several works separating the process into salience detection and generation components (Barzilay and McKeown, 2005; Li et al., 2018; Ernst et al., 2022). These works focused on optimizing each component as part of a fully-automatic summarization process in order to improve the overall performance of the model. In contrast, our work uses this modularity to not only improve overall system output, but to also give more control to the user over each step in the summarization process.

Our objective in this paper is to promote such a human-involved approach to summarization, allowing to better tailor the eventual output to real-world user needs, and to synergize the efficiency of the computer with the quality of the human (Hoc, 2000; Pacaux-Lemoine et al., 2017; Flemisch et al., 2019).

it is crucial to prioritize and direct human efforts toward more "suspicious" outputs from LLMs

we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels

LLM annotators and human annotators should not be treated the same, and annotation tools should carefully design their data models and workflows to accommodate both types of annotators

it is advisable to either mask any confidential information or only use in-house LLMs

it is recommended that the format of a prompt be similar to the one used in training as some LLMs have different prompt format than the others

the selection of label options may work better if it is similar to common options for given tasks, such as [positive, neutral, negative] > [super positive, positive, ..., negative] for sentiment classification

designing an annotation task and a prompt similar to more widely used and standardized NLP tasks is beneficial

errors encountered during API calls are handled in two ways: handle within our system or delegate to users. We handle known LLM API errors that can be solved by user-side intervention. This would be in cases such as a Timeout or RateLimitError in OpenAI models

errors such as APIConnectionError in OpenAI models occur because of an issue with the LLM API server itself and requires intervention from OpenAI.

While MEGAnno+ is designed to support any open-source LLM or commercial LLM APIs, in this work, we only demonstrate OpenAI Completion models for clarity and brevity.

Data Model MEGAnno+ extends MEGAnno's data model where data Record, Label, Annotation, Metadata (e.g., text embedding or confidence score) are persisted in the service database along with the task Schema.

To implement our system as an extension to MEGAnno (Zhang et al., 2022), an in-notebook exploratory annotation tool.

MEGAnno+ is designed to provide a convenient and robust workflow for users to utilize LLMs in text annotation. To use our tool, users operate within their Jupyter notebook (Kluyver et al., 2016) with the MEGAnno+ client installed.

LLM annotators and human annotators should not be treated the same, and annotation tools should carefully design their data models and workflows to accommodate both types of annotators.

we go beyond using LLMs to assist annotation for human annotators or to replace human annotators. Rather, MEGAnno+ advocates for a collaboration between humans and LLMs with our dedicated system design and annotation-verification workflows.

Despite these advancements, it is essential to acknowledge that LLMs have limitations, necessitating human intervention in the data annotation process. One challenge is that the performance of LLMs varies extensively across different tasks, datasets, and labels. LLMs often struggle to comprehend subtle nuances or contexts in natural language, making involvement of humans with social and cultural understanding or domain expertise crucial.

Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels.

Valarie A Zeithaml and William L Fuerst. 1983. Age differences in response to grocery store price information. Journal of consumer affairs 17, 2 (1983), 402–420.

Mary E Sesto, Curtis B Irwin, Karen B Chen, Amrish O Chourasia, and Douglas A Wiegmann. 2012. Effect of touch screen button size and spacing on touch characteristics of users with and without disabilities. Human Factors: The Journal of the Human Factors and Ergonomics Society 54, 3 (2012), 425–436.

Zhao Xia Jin, Tom Plocher, and Liana Kiff. 2007. Touch screen user interfaces for older adults: button size and spacing. In Universal acess in human computer interaction. coping with diversity. Springer, 933–941.

Robin Brewer, Raymundo Cornejo Garcia, Tedmond Schwaba, Darren Gergle, and Anne Marie Piper. 2016. Exploring Traditional Phones as an E-Mail Interface for Older Adults. ACM Transactions on Accessible Computing (TACCESS) 8, 2 (2016), 6.

Janan Al-Awar Smither and Curt C Braun. 1994. Technology and older adults: Factors affecting the adoption of automatic teller machines. The Journal of General Psychology 121, 4 (1994), 381–389.

Wiktoria Wilkowska and Martina Ziefle. 2009. Which factors form older adults' acceptance of mobile information and communication technologies? Springer.

Kerryellen G Vroman, Sajay Arthanat, and Catherine Lysack. 2015. "Who over 65 is online?" Older adults' dispositions toward information communication technology. Computers in Human Behavior 43 (2015), 156–166.

Phil Turner, Susan Turner, and Guy Van de Walle. 2007. How older people account for their experiences with interactive technology. Behaviour & Information Technology 26, 4 (2007), 287–296.

Hironobu Takagi, Akihiro Kosugi, Tatsuya Ishihara, and Kentarou Fukuda. 2014. Remote IT education for senior citizens. In Proceedings of the 11th Web for All Conference. ACM, 41.

Karen Renaud and Judy Van Biljon. 2008. Predicting technology acceptance and adoption by the elderly: a qualitative study. In Proceedings of the 2008 annual research conference of the South African Institute of Computer Scientists and Information Technologists on IT research in developing countries: riding the wave of technology. ACM, 210–219.

Chee Wei Phang, Juliana Sutanto, Atreyi Kankanhalli, Yan Li, Bernard CY Tan, and Hock-Hai Teo. 2006. Senior citizens' acceptance of information systems: A study in the context of e-government services. Engineering Management, IEEE Transactions on 53, 4 (2006), 555–569.

Bjorn Niehaves and Ralf Plattfaut. 2014. Internet adoption by the elderly: employing IS technology acceptance theories for understanding the age-related digital divide. European Journal of Information Systems 23, 6 (2014), 708–726.

HH Nap and HP de Greef. 2010. Self-efficacy & stress in senior computer interaction. In Proceedings of the 28th Annual European Conference on Cognitive Ergonomics. ACM, 227–230.

Michael G Morris and Viswanath Venkatesh. 2000. Age differences in technology adoption decisions: Implications for a changing work force. Personnel psychology 53, 2 (2000), 375–403.

Tracy L Mitzner, Wendy A Rogers, Arthur D Fisk, Walter R Boot, Neil Charness, Sara J Czaja, and Joseph Sharit. 2014. Predicting older adults' perceptions about a computer system designed for seniors. Universal Access in the Information Society (2014), 1–10.