You have to have people that have the ability to rethink the workflow at a scale that AI can execute, versus at a scale that humans can execute.

highlights are incorporated into the input text with special markups, <extra_id_1> and <extra_id_2>, marking the beginning and end of each highlighted span, respectively. In our configuration, we set the maximum input length to 4096 and the maximum target length to 400. A greedy decoding strategy was used in order to optimize the decoding speed.

Our approach locates the longest common subsequence (LCS) between the lemmas of each input sentence and each summary sentence, followed by several heuristics to filter out irrelevant LCSs

For the summarization model, we used a BARTlarge model (Lewis et al., 2019) trained on the CNN/Daily Mail dataset (Hermann et al., 2015), selected for its noticeable popularity.

For the initial auto-consolidation, we deploy an available Controlled Text Reduction model (Slobodkin et al., 2023), which is a Flan-T5large model (Chung et al., 2022), finetuned on the highlights-focused CTR dataset.

we deploy the ExtractiveSummarizer model from the TransformerSum library. The model, a RoBERTabase (Liu et al., 2019) trained on the CNN/DailyMail summarization dataset (Hermann et al., 2015), operates as a binary classifier.

errors encountered during API calls are handled in two ways: handle within our system or delegate to users. We handle known LLM API errors that can be solved by user-side intervention. This would be in cases such as a Timeout or RateLimitError in OpenAI models

errors such as APIConnectionError in OpenAI models occur because of an issue with the LLM API server itself and requires intervention from OpenAI.

While MEGAnno+ is designed to support any open-source LLM or commercial LLM APIs, in this work, we only demonstrate OpenAI Completion models for clarity and brevity.

Data Model MEGAnno+ extends MEGAnno's data model where data Record, Label, Annotation, Metadata (e.g., text embedding or confidence score) are persisted in the service database along with the task Schema.

To implement our system as an extension to MEGAnno (Zhang et al., 2022), an in-notebook exploratory annotation tool.

MEGAnno+ is designed to provide a convenient and robust workflow for users to utilize LLMs in text annotation. To use our tool, users operate within their Jupyter notebook (Kluyver et al., 2016) with the MEGAnno+ client installed.