it is crucial to prioritize and direct human efforts toward more "suspicious" outputs from LLMs

we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels

LLM annotators and human annotators should not be treated the same, and annotation tools should carefully design their data models and workflows to accommodate both types of annotators

it is advisable to either mask any confidential information or only use in-house LLMs

it is recommended that the format of a prompt be similar to the one used in training as some LLMs have different prompt format than the others

the selection of label options may work better if it is similar to common options for given tasks, such as [positive, neutral, negative] > [super positive, positive, ..., negative] for sentiment classification

designing an annotation task and a prompt similar to more widely used and standardized NLP tasks is beneficial

errors encountered during API calls are handled in two ways: handle within our system or delegate to users. We handle known LLM API errors that can be solved by user-side intervention. This would be in cases such as a Timeout or RateLimitError in OpenAI models

errors such as APIConnectionError in OpenAI models occur because of an issue with the LLM API server itself and requires intervention from OpenAI.

While MEGAnno+ is designed to support any open-source LLM or commercial LLM APIs, in this work, we only demonstrate OpenAI Completion models for clarity and brevity.

Data Model MEGAnno+ extends MEGAnno's data model where data Record, Label, Annotation, Metadata (e.g., text embedding or confidence score) are persisted in the service database along with the task Schema.

To implement our system as an extension to MEGAnno (Zhang et al., 2022), an in-notebook exploratory annotation tool.

MEGAnno+ is designed to provide a convenient and robust workflow for users to utilize LLMs in text annotation. To use our tool, users operate within their Jupyter notebook (Kluyver et al., 2016) with the MEGAnno+ client installed.

LLM annotators and human annotators should not be treated the same, and annotation tools should carefully design their data models and workflows to accommodate both types of annotators.

we go beyond using LLMs to assist annotation for human annotators or to replace human annotators. Rather, MEGAnno+ advocates for a collaboration between humans and LLMs with our dedicated system design and annotation-verification workflows.

Despite these advancements, it is essential to acknowledge that LLMs have limitations, necessitating human intervention in the data annotation process. One challenge is that the performance of LLMs varies extensively across different tasks, datasets, and labels. LLMs often struggle to comprehend subtle nuances or contexts in natural language, making involvement of humans with social and cultural understanding or domain expertise crucial.

Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels.