wasting compute on semantically empty tokens.Unpadding avoids this inefficiency by removingpadding tokens, concatenating all sequencesfrom a minibatch into a single sequence, andprocessing it as a batch of one. Prior unpaddingimplementations unpad and repad sequencesinternally for different model layers, wastingcompute and memory bandwidth. We use FlashAttention’s variable length attention and RoPEimplementations, allowing jagged attention masksand RoPE applications on one unpadded sequence.ModernBERT unpads inputs before the tokenembedding layer and optionally repads modeloutputs leading to a 10-to-20 percent performanceimprovement over other unpadding methods.

Alternating Attention Following recent work onefficient long context models (Gemma et al., 2024),attention layers in ModernBERT alternate betweenglobal attention, where every token within a se-quence attends to every other token, and local atten-tion, where tokens only attend to each other withina small sliding window (Beltagy et al., 2020). InModernBERT, every third layer employs globalattention with a RoPE theta of 160,000 and theremaining layers use a 128 token, local sliding win-dow attention with a RoPE theta of 10,000

Bias Terms Following (Dayma et al., 2021), wedisable bias terms in all linear layers except for the1FlexBERT is built on top of a revised Mo-saicBERT (Portes et al., 2023) codebase.final decoder linear layer2. We also disable all biasterms in Layer Norms (Xu et al., 2019). These twochanges allow us to spend more of our parameterbudget in linear layers

Positional Embeddings We use rotary posi-tional embeddings (RoPE) (Su et al., 2024) insteadof absolute positional embeddings. This choice ismotivated by the proven performance of RoPE inshort- and long-context language models (Blacket al., 2022; Dubey et al., 2024; Gemma et al.,2024), efficient implementations in most frame-works, and ease of context extension

Activation We adopt GeGLU (Shazeer, 2020),a Gated-Linear Units (GLU)-based (Dauphin et al.,2017) activation function built on top of the origi-nal BERT’s GeLU (Hendrycks and Gimpel, 2016)activation function. This is in line with recent workshowing consistent empirical improvements whenusing GLU variants (Shazeer, 2020; Geiping andGoldstein, 2023).

sub-optimal documentsin terms of scope and diversity, large-scale websearches (Piktus et al., 2021; Komeili et al., 2022)are integrated as a strategic extension of RAG.Specifically, the inputs are rewritten into queriescomposed of keywords by ChatGPT to mimic thedaily usage of search engine. The prompt forrewriting is shown in Appendix A. In CRAG,a public and accessible commercial web searchAPI is adopted to generate a series of URL linksfor every query. 3 Considering that knowledgefrom large-scale web searches could introducebiases or unreliable information, authoritative andregulated web pages like Wikipedia are preferred,which can significantly help mitigate these issues.Moreover, we utilize the URL links to navigateweb pages, transcribe their content, and employ thesame knowledge refinement method as Section 4.4to derive the relevant web knowledge, namelyexternal knowledge.

Therefore, it is extremely importantto seek complementary external knowledge ifthe retrieved results are all assumed irrelevant,and we consider a system that knows what itdoesn’t know and what it cannot answer to bemore intelligent than one that clings to limitedknowledge and is incapable of seeking externalknowledge

4.4 Knowledge RefinementGiven a retrieved relevant document, a decompose-then-recompose knowledge refinement methodis designed to further extract the most criticalknowledge strips in it. To obtain fine-grainedretrieval results, we segmented the retrieved resultsinto internal strips. If a retrieved result is as short asone or two sentences, it is regarded as an individualstrip, otherwise, retrieval documents are required tobe split into smaller units which generally consistof a few sentences according to the total length.The scale is assumed to include an independentpiece of information, and the filtering is based onthe segments. Then, the retrieval evaluator fine-tuned in Section 4.2 is employed to calculate therelevance score of each knowledge strip. Basedon these scores, irrelevant knowledge strips arefiltered out, while relevant ones are recomposed viaconcatenation in order, namely internal knowledge

Incorrect Besides, a retrieval is assumedIncorrect when the confidence scores of allretrieved documents are below the lower threshold.This indicates that all retrieved documents areconsidered irrelevant, which are unhelpful forgeneration. Once the knowledge from the retrievalresults is judged to be inaccurate, it is unwise tostill get stuck in it, which is likely to result infabricated facts. Therefore, we need to seek newsources of knowledge for correction. Here, websearch is introduced to search from the Internet aselaborated in Section 4.5. This corrective actionhelps overcome the embarrassing challenge whereno reliable knowledge can be referred to.

Correct Here, a retrieval is assumed Correctwhen the confidence score of at least one retrieveddocument is higher than the upper threshold. Ifso, it means that there are relevant documents inthe retrieved results, and the knowledge from theretrieval results is supposed to be more reliable andaccurate. However, even if a relevant document canbe found, there is inevitably some noisy knowledgestrips in this document. To extract the mostcritical knowledge strips within this document, aknowledge refinement method is further designedwhich will be elaborated in Section 4.4.

Based on the aforementionedconfidence score for each retrieved document, threetypes of actions are designed and triggered accord-ingly where the upper and lower thresholds are set.If the confidence score is higher than the upperthreshold, the retrieved document is identified asCorrect, while identified as Incorrect if belowthe lower threshold. Otherwise, a more soft andintermediate action, i.e., Ambiguous is executed.Each retrieved document is conducted individuallyand integrated eventually.

he relevance signalsfor fine-tuning the evaluator can be collected fromthe existing datasets. For example, PopQA (Mallenet al., 2023) provides the golden subject wiki titlefrom wikipedia for each question. We can use thatto track a not 100% relevant but rather high-qualitypassage. We utilized that as the relevance signalsfor fine-tuning the retrieval evaluator.2 On the otherhand, the negative samples for fine-tuning wereall randomly sampled from the retrieval results,which are rather similar to the input query but2https://huggingface.co/datasets/akariasai/PopQAnot relevant. More details about this fine-tuningstep can be referred to in Appendix B.3. Forevery question, there are generally 10 documentsretrieved. The question is concatenated with eachsingle document as the input, and the evaluatorpredicts the relevance score for each question-document pair individually. We also tried to promptChatGPT to identify the retrieval relevance forcomparison, but it underperforms as elaborated inSection 5.5. Based on these calculated relevancescores, a final judgment is made as to whetherthe retrieval is correct or not associated with theaction trigger. In our proposed framework, theretrieval quality is evaluated at a relatively lowcost without the need to have access to large andexpensive LLMs. Compared with the critic modelof Self-RAG (Asai et al., 2024) that instruction-tuned LLaMA-2 (7B), the evaluator designed inCRAG demonstrates the advantages of being quitelightweight (0.77B).

To ensure all experimental results werecomparable with Self-RAG (Asai et al., 2024), thesame retrieval results through Contriever (Izacardet al., 2022) provided by Self-RAG were alsoadopted in our experiments

Specifically, T5-large (Raffelet al., 2020) is adopted for initializing the retrievalevaluator and fine-tuned

Our objective is to correct the retrieved documentsif they are irrelevant

Figure 2 and Algorithm 1 present an overviewof CRAG at inference, which designs correctivestrategies to improve the robustness of generation.Given an input query and the retrieved documentsfrom any retriever, a lightweight retrieval evaluatoris constructed to estimate the relevance scoreof retrieved documents to the input query (Sec-tion 4.2). The relevance score is quantified into atotal of three confidence degrees and then triggeredthe corresponding actions: {Correct, Incorrect,Ambiguous} (Section 4.3). If the action Correctis triggered, the retrieved documents will be re-fined into more precise knowledge strips. Thisrefinement operation involves knowledge decom-position, filter, and recomposition (Section 4.4).If the action Incorrect is triggered, the retrieveddocuments will be discarded. Instead, web searchesare resorted to and regarded as complementaryknowledge sources for corrections (Section 4.5).Eventually, when it cannot confidently make acorrect or incorrect judgment, a soft and balancedaction Ambiguous which combines both of them istriggered. After optimizing the retrieval results, anarbitrary generative model can be adopted

Besides, a decompose-then-recompose algorithm is designed for retrieveddocuments to selectively focus on key infor-mation and filter out irrelevant information inthem

Specifically, a lightweightretrieval evaluator is designed to assess theoverall quality of retrieved documents for aquery, returning a confidence degree basedon which different knowledge retrieval ac-tions can be triggered.

Since retrieval fromstatic and limited corpora can only return sub-optimal documents, large-scale web searchesare utilized as an extension for augmenting theretrieval results.

Al-though retrieval-augmented generation (RAG)is a practicable complement to LLMs, it reliesheavily on the relevance of retrieved docu-ments, raising concerns about how the modelbehaves if retrieval goes wrong

he training of M3-Embedding poses a signifi-cant challenge. In our work, the following technicalcontributions are made to optimize the embeddingquality. Firstly, we propose a novel self knowl-edge distillation framework, where the multipleretrieval functionalities can be jointly learned andmutually reinforced. In M3-Embedding, the [CLS]embedding is used for dense retrieval, while embed-dings from other tokens are used for sparse retrievaland multi-vector retrieval. Based on the principleof ensemble learning (B ̈uhlmann, 2012), such het-erogenous predictors can be combined as a strongerpredictor. Thus, we integrate the relevance scoresfrom different retrieval functions as the teachersignal, which is used to enhance the learning pro-cess via knowledge distillation. Secondly, we op-timize the batching strategy to achieve a largebatch size and high training throughput, which sub-stantially contributes to the discriminativeness ofembeddings. Last but not least, we perform exten-sive and high-quality data curation. Our datasetincludes three sources: 1) the extraction of unsuper-vised data from massive multi-lingual corpora, 2)the integration of closely related supervised data, 3)the synthesization of scarce training data. The threedata sources are complement to each other and ap-plied to different training stages, which lays a solidfoundation for the versatile text embeddings.

espite the widespread popularity of text em-beddings, the existing methods are still limited inversatility. First of all, most of the embedding mod-els are tailored only for English, leaving few viableoptions for the other languages. Secondly, the exist-ing embedding models are usually trained for onesingle retrieval functionality. However, typical IRsystems call for the compound workflow of multi-ple retrieval methods. Thirdly, it is challenging totrain a competitive long-document retriever due tothe overwhelming training cost, where most of theembedding models can only support short inputs.

One com-mon form of embedding-based IR application is†. Co-first author∗. Corresponding authors1. The model, code, and data is publicly available athttps://github.com/FlagOpen/FlagEmbedding.BGE M3-EmbeddingMulti-LingualCross-Lingual100+ LanguagesSparse RetrievalMulti-Vec RetrievalDense RetrievalPassage-LevelDoc-Level (≤8192)Sentence-LevelMulti-Linguality Multi-Functionality Multi-GranularityM3-EmbeddingMulti-LingualCross-Lingual100+ LanguagesSparse RetrievalMulti-Vec RetrievalDense RetrievalPassage-LevelDoc-Level (≤8192)Sentence-LevelMulti-Linguality Multi-Functionality Multi-GranularityFigure 1: Characters of M3-Embedding.dense retrieval, where relevant answers to the querycan be retrieved based on the embedding similarity(Karpukhin et al., 2020; Xiong et al., 2020; Nee-lakantan et al., 2022; Wang et al., 2022; Xiao et al.,2023). Besides, the embedding model can also beapplied to other IR tasks, such as multi-vector re-trieval where the fine-grained relevance betweenquery and document is computed based on the in-teraction score of multiple embeddings (Khattaband Zaharia, 2020), and sparse or lexical retrievalwhere the importance of each term is estimated byits output embedding (Gao et al., 2021a; Lin andMa, 2021; Dai and Callan, 2020)

ArmoRM. Wang et al. (2024b) argue that cur-rent reward models often conflate different objec-tives, making it difficult to discern which aspectsof the input data influence their scoring. To ad-dress this, they proposed the ArmoRM (AbsoluteRating Multi-Objective Reward Model). As illus-trated in Figure 14, the model processes a contextand multiple candidate responses, evaluating themacross interpretable dimensions such as honesty,safety, verbosity, and relevance. Each dimensionis assessed by a dedicated sub-model that gener-ates individual scores. These scores are then dy-namically weighted by a gating network, whichadapts to the context and produces a final rewardscore used as feedback for reinforcement learn-ing. This mixture-of-experts approach effectivelyseparates the objectives, allowing the scores to bemore clearly attributed to specific input featuresor goals, thus improving both interpretability andtransparency

Out-of-distribution (OOD) issues present a signif-icant challenge in reward modeling, particularlywhen the reward model and the large languagemodel (LLM) are trained independently. This sep-aration can lead to inconsistencies in the knowl-edge and decision-making frameworks of the twomodels, potentially causing the reward model toencounter unfamiliar scenarios or fail to generalizeeffectively. Addressing OOD challenges is criticalfor ensuring that reward models (RMs) performreliably across diverse inputs

These include addressing out-of-distribution issuesbetween the trained reward models and the alignedLLMs, ensuring the interpretability of the modelfor humans, and maintaining safety and evaluationbenchmarks to train robust reward models. In thissection, we discuss recent works that tackle thesechallenges and provide strategies for overcomingthem

Magpie. Xu et al. (2024b) introduce a self-synthesis method that leverages the autoregressivenature of aligned LLMs. By utilizing predefinedtemplates as prompts, the model autonomously gen-erates user queries and corresponding responses,eliminating the need for manual intervention orinitial seed questions. Specifically, as shown in Fig-ure 4, aligned LLMs (e.g., Llama-3-Instruct model)is employed to synthesize 4 million instruction-response pairs, subsequently filtering the dataset toretain 300,000 high-quality pairs. These pairs werethen used to fine-tune the Llama-3-8B-Base model.Remarkably, the fine-tuned model achieved per-formance comparable to the official Llama-3-8B-Instruct model, which had undergone training on10 million examples through supervised fine-tuningand reinforcement learning with human feedback.Besides, models fine-tuned with Magpie excelledon alignment benchmarks such as AlpacaEval, sur-passing models trained on other open datasets andpreference optimization methods.

RLHF that leverages AI systems—often more pow-erful or specialized LLMs (e.g., GPT-4 (OpenAI,2024a))—to provide feedback on the outputs ofthe LLM being trained. This approach providesbenefits such as scalability, consistency, and costefficiency while minimizing reliance on humanevaluators. Below, we explore several methodsfor substituting human feedback with AI feedbackin reinforcement learning, highlighting approaches:(1) Distilling AI Feedback to Train Reward Model,(2) Prompting LLMs As a Reward Function, and(3) Self-Rewarding.

(1) Rewarding: In this step, the LLM generatesmultiple outputs in response to a given instruction.Each output is then passed through the trained re-ward model, which assigns a scalar score that ap-proximates human preferences.(2) Policy Optimization: In this step, the LLM isfine-tuned by adjusting its parameters to maximizethe predicted reward, using the Proximal Policy Op-timization (PPO) (Schulman et al., 2017) or TrustRegion Policy Optimization (TRPO) (Schulman,2015) algorithm.

TÜLU-V2-mix (Ivison et al., 2023). TÜLU-V2-mix is designed to enhance instruction-followingcapabilities in large language models, offering adiverse dataset that improves the model’s general-ization and execution abilities across multi-domaintasks. It covers a wide range of tasks, includingquestion answering, code generation, translation,and multi-turn conversations, with a strong em-phasis on multilingual adaptability and handlingcomplex real-world scenarios. Skywork-Reward,on the other hand, is designed to align models withhuman preferences using preference pairs, helpingmodels learn to generate user-preferred responses,such as fluent and coherent text. While TÜLU-V2-mix excels in generalization across a wide range oftasks, Skywork-Reward specializes in optimizinguser-centric outputs. Together, they address com-plementary goals for advancing language modelcapabilities

(1) Collecting Human Feedback to TrainReward Model, where human evaluators providefeedback on the LLM’s outputs by scoring or rank-ing responses based on factors such as quality andrelevance. This feedback is then used to train a re-ward model that predicts the quality of the outputsand serves as the reward function in the RL process;and (2) Preference Optimization Using HumanFeedback, where the trained reward model guidesthe optimization of the LLM’s outputs to maximizepredicted rewards, aligning the LLM’s behaviorwith human preferences. Below, we will illustratethese two components via recent research studies

The post-training process for aligning Llama 3with human feedback involves six rounds of iter-ative refinement. Each round includes supervisedfine-tuning (SFT) followed by DPO, with the fi-nal model being an average of the outputs from allrounds. For each round, a reward model (RM) istrained on newly collected preference annotationdata, targeting a wide range of capabilities builtupon the pre-trained checkpoint. After SFT, DPOis applied to further optimize the SFT models, us-ing recent preference data batches obtained fromthe best-performing models of previous rounds. Toenhance the stability of DPO training, two key ad-justments are implemented: masking out format-ting tokens in the DPO loss and introducing reg-ularization via an NLL (negative log-likelihood)loss.

Gemini implements a post-training process thatutilizes an optimized feedback loop, collectinghuman-AI interactions to drive continuous improve-ment in key performance areas. During the post-training’s RLHF phase, an iterative approach isadopted wherein reinforcement learning (RL) in-crementally enhances the reward model (RM). Con-currently, the RM undergoes continuous refinementthrough systematic evaluation and data collection.This dynamic interplay promotes ongoing advance-ment in both RL and RM, leading to progressivelyimproved performance over time

GPT-4 leverages RLHF methods, as outlinedin InstructGPT (Ouyang et al., 2022) which wehave describe in Sec 3.1, in the post-training align-ment stage. To steer the models more effectivelytowards appropriate refusals at a finer level, theauthors further use a zero-shot GPT-4 classifier asthe rule-based reward model (RBRM). This RBRMprovides an additional reward signal to the GPT-4policy model during PPO fine-tuning on a subsetof training prompts. The RBRM takes a prompt(optional), the policy model’s output, and a human-written rubric (e.g., a set of rules in multiple-choicestyle) as input, then classifies the output accord-ing to the rubric. Through this approach, GPT-4is rewarded for refusing harmful content and forappropriately responding to known-safe prompts.

1. Traditional RL approaches, such as Reinforce-ment Learning from Human Feedback (RLHF)and Reinforcement Learning from AI Feedback(RLAIF). These methods require training a rewardmodel and involve a complex and often unstableprocess, using algorithms like Proximal Policy Op-timization (PPO) (Schulman et al., 2017) to opti-mize the policy model. Models like InstructGPT(Ouyang et al., 2022), GPT-4 (OpenAI, 2023), andClaude 3 (Anthropic, 2024) follow this approach.2. Simplified approaches, such as Direct Prefer-ence Optimization (DPO) (Rafailov et al., 2024)and Reward-aware Preference Optimization (RPO)(Adler et al., 2024). These methods discard thereward model, offering a stable, performant, andcomputationally efficient solution. Models likeLlama 3 (Dubey et al., 2024), Qwen 2 (Yang et al.,2024a), and Nemotron-4 340B (Adler et al., 2024)follow this approach. In this section, we providea detailed description of each model, starting witha brief overview of these RL enhanced LLMs andfollowed by an explanation of how RL is applied intheir post-training process. An overview of these

Step 1: Collect comparison data, and traina reward model. Ouyang et al. (2022) collectsa dataset of comparisons between outputs of theinstruction-tuned model, where labelers indicatewhich output they prefer for a given input. Then,the collected dataset is used to train a reward model(RM) to predict the human-preferred output.Step 2: Optimize a policy against the rewardmodel using PPO. Ouyang et al. (2022) leveragesthe output of the RM as a scalar reward, and fine-tunes the instruction-tuned model to optimize thisreward using the PPO algorithm (Schulman et al.,2017).

e have outlined the general framework of RLabove; now we will delve into the process of fine-tuning LLMs using RL. This approach aims to alignLLMs with desired behaviors, enhance their per-formance, and ensure that their outputs are botheffective and dependable.

or the RM architecture, we use pre-trained transformer-based language models with the last unem-bedding layer removed and add an additional linear layer to the final transformer layer. Given anytext, the reward model will assign a scalar reward value to the last token, and the larger the rewardvalue, the better the sample. Following Stiennon et al. [25], training reward models often involvesutilizing a dataset comprised of paired comparisons between two responses generated for the sameinput. The modeling loss for each pair of preferred and dispreferred samples is:

PPO workflow, depicting the sequential steps in the algorithm’s execution. The processbegins with sampling from the environment, followed by the application of GAE for improvedadvantage approximation. The diagram then illustrates the computation of various loss functionsemployed in PPO, signifying the iterative nature of the learning process and the policy updatesderived from these losses.

In this report, we carefully dissect the framework of RLHF and discuss the entire process thatdetermines the success of the algorithm’s training. We explored how the quality of the reward modelaffects the final result of the policy model. We find that the quality of the reward model directlydetermines the upper bound of the policy model, and designing an appropriate PPO algorithm is crucialfor RLHF’s successful training. Moreover, accurate code implementation matters in deep policy(practice makes perfect). Therefore, we have conducted in-depth evaluations of the inner workingsof PPO algorithm to study how code-level and theory-level optimizations change agent trainingdynamics. We propose to monitor the PPO training process by using action space modeling metricsderived from the policy model, such as perplexity, response length, and KL divergence betweenthe policy model and the SFT model. These metrics are more informative of the training stabilitythan the values of response reward and loss functions. Based on these observations, we identify thepolicy constraints in the PPO algorithm as the key factor to achieve consistent alignment with humanpreferences. After extensive comparative experiments with various possible implementations of PPOframework, we finally introduce a preferable policy optimization algorithm named PPO-max, whichincorporates the collection of effective and essential implementations, and is carefully calibratedto avoid interference among them. PPO-max alleviates the instability of vanilla PPO training andenables longer training steps with a larger training corpus. We evaluate PPO-max on 7B and 13BSFT models, demonstrating comparable alignment performance with ChatGPT

Q1: How do the quality, dimensionality, andgranularity of LLM-generated feedback (C1–C4 inTable 2) influence preference learning?Q2: What impact does each dimension have inthe case of multi-dimensional feedback?Q3: How much can DPO enhance the quality ofsummaries compared to SFT variants

Coarse-grained evaluation uses a Likert scale (1–5), butthese subjective scores often skew toward higherratings due to a lack of grounding (Wang et al.,2023; Liu et al., 2023a). In contrast, fine-grainedevaluation assesses at the sentence or key-fact level,measuring faithfulness, completeness, and concise-ness through factual sentence proportions and key-fact coverage, yielding percentage scores bettercorrelate with human feedback (Song et al., 2024).Thus, all summary–document pairs are subjectedto LLM-based summarization evaluation for eachconfiguration. 125K summary-document pairs re-main after excluding failed and special cases.2More details including evaluation prompts, method-ologies, and data statistics are in Appendix A.

Feedback Dimensionality (C2 vs. C3): The sim-plest way to gather feedback is to assess the qualityof the summary with a single score on a 1–5 Likert2We exclude cases where the document length exceeds themodel’s capacity (8K tokens by Llama3 on a single NVIDIAH100 GPU), as well as instances of erroneous feedback fromthe LLMs, such as incomplete or nonsensical responses.scale (Wang et al., 2023). However, it overlookskey multi-dimensional aspects of summary quality,such as faithfulness, completeness, and concise-ness (Lee et al., 2024). Therefore, a more advancedapproach involves conducting a multi-dimensionalevaluation using LLMs across these three dimen-sions, yielding a separate score for each (Zhonget al., 2022; Liu et al., 2023a).

Feedback Quality (C1 vs. C2): The quality ofgenerated feedback plays a pivotal role in prefer-ence learning. To assess the importance of feed-back quality, we adjust the capacity of the selectedLLMs for feedback generation. We use two open-source LLMs of different sizes: Llama3-8b-instructfor low-quality feedback and Llama3-70b-instructfor high-quality feedback, respectively

Feed-back is generated through LLM-based summaryevaluations using four configurations, adjustingthe quality (low vs. high), dimensionality (singlevs. multi-dimensional), and granularity (coarse- vs.fine-grained) of the feedback.

In this work, we integrate automated evaluationinto preference learning, enabling large-scale, fine-grained feedback that addresses three key align-ment dimensions of text summarization: faithful-ness, completeness, and conciseness

(1) We create andrelease FeedSum, the first large-scale summariza-tion dataset for preference learning, featuring highdiversity in inputs, summaries, and feedback; (2)We examine how different configurations of LLM-generated feedback impact preference learning, andthe importance of feedback quality, dimensional-ity, and granularity. (3) We examine the alignmenttrade-off associated with relying on a single dimen-sion for preference learning. (4) We compare theeffectiveness of DPO and SFT variants. (5) Werelease SummLlama3-8b, which outperforms thenearly 10x larger Llama3-70b-instruct in the threehuman-preferred dimensions.

Each setup generates 125K pairs of in-put documents and summaries, accompanied bydistinct LLM-generated feedback responses. In

from LLM feedback, focusing on three key openquestions: (Q1) The impact of these three factorson the effectiveness of preference learning; (Q2)An analysis on the effectiveness of each dimen-sion in multi-dimensional feedback; and (Q3) Acomparison between two approaches for utilizingLLM-generated feedback: supervised fine-tuning(SFT) and direct preference optimization (DPO).

To obtain summaries with vary-ing levels of quality, we generate them using 13different language models, including 3 non-LLMs(e.g., Bart), 7 open-source LLMs (e.g., Llama3),and 3 proprietary LLMs (e.g., GPT-4-turbo). Forsummary generation, these models are applied toa diverse range of input documents, spanning bothshort and lengthy texts, including dialogue and non-dialogue formats, and across 7 distinct domains.

Figure 1 illustrates our com-plete pipeline for learning from LLM-generatedfeedback, consisting of data sourcing, feedbackgeneration, and preference learning

The effectiveness of LLM feedback on prefer-ence learning can vary based on three factors, in-cluding the quality of the feedback (e.g., low vs.high), its dimensionality (e.g., single vs. multiple),and its level of granularity in scoring (e.g., coarsevs. fine)

While large language models (LLMs)have greatly improved the coherence and fluencyof summaries (Liu et al., 2023a), persistent issuesremain, such as unfaithful statements (hallucina-tions), omission of key information (low complete-* Corresponding Author.† This work is conducted independently and is not relatedto the author(s)’ position at Hyperconnect and Amazon.ness), and verbosity (low conciseness) in the sum-maries (Lee et al., 2024; Song et al., 2024)

Our approach shifts focus to the relatively unex-plored area of leveraging LLM-generated feedbackto enhance summary quality, whereas most exist-ing research in summarization has primarily con-centrated on using LLMs to evaluate summaries.(Wan et al., 2024; Tang et al., 2024a; Song et al.,2024). Specifically, our goal is to produce human-preferred summaries by exploiting LLM feedbackwith respect to the three core dimensions, namelyfaithfulness, ensuring summaries are consistentwith original documents; completeness, encompass-ing all key-facts1; and conciseness, maintaining asuccinct and focused summary.

Developing effective text summarizers remainsa challenge due to issues like hallucinations,key information omissions, and verbosity inLLM-generated summaries. This work ex-plores using LLM-generated feedback to im-prove summary quality by aligning the sum-maries with human preferences for faithful-ness, completeness, and conciseness. We in-troduce FeedSum, a large-scale dataset con-taining multi-dimensional LLM feedback onsummaries of varying quality across diversedomains. Our experiments show how feed-back quality, dimensionality, and granularityinfluence preference learning, revealing thathigh-quality, multi-dimensional, fine-grainedfeedback significantly improves summary gen-eration. We also compare two methods for us-ing this feedback: supervised fine-tuning anddirect preference optimization. Finally, we in-troduce SummLlama3-8b, a model that out-performs the nearly 10x larger Llama3-70b-instruct in generating human-preferred sum-maries, demonstrating that smaller modelscan achieve superior performance with appro-priate training. The full dataset will be re-leased soon. The SummLlama3-8B model isnow available at https://huggingface.co/DISLab/SummLlama3-8B

Figure 9(b) presents the results of of PaLM 2-L as the scorer LLM with the following options ofinitial instructions: (1) “Let’s solve the problem.”; (2) the empty string; or (3) “Let’s think stepby step.”. We notice that the performance differs much more with different initial instructions,especially at the beginning of the optimization. Specifically, starting from (1) leads to better generatedinstructions than (2) in the first 30 steps, while the instructions optimized from both (1) and (2)are worse than (3) throughout. A similar observation holds when using PaLM 2-L as scorer andgpt-3.5-turbo as optimizer for BBH tasks, by comparing the results starting from the emptystring (Appendix E.2) and from “Let’s solve the problem.” (Appendix E.3). Taking a closer look intothe optimization process of (2), we find that although both “solve the problem” and “step by step”show up in generated instructions at Step 5, it takes the optimizer LLM more steps to get rid of worseinstructions presented in the meta-prompt when starting from instructions with lower accuracies.Therefore, one direction for future work is to accelerate convergence from weaker starting points.

The number of generated instructions per step. Computing a mini-batch of gradients reducesthe variance of a stochastic gradient descent procedure. Similarly, generating multiple instructionsin each step improves the optimization stability with LLMs. On the other hand, to achieve betterperformance with a fixed budget for the number of instructions to evaluate, the number of per-stepinstructions should not be too large, so as to allow more optimization steps to incorporate richerinformation of past instructions with their accuracies. Taking both aspects into consideration, Figure 8compares the optimization performance of sampling 1 / 2 / 4 / 8 (default) / 16 instructions in eachstep, showing that sampling 8 instructions at each step overall achieves the best performance

Meta-prompt design. The meta-prompt design is crucial in achieving good prompt optimizationperformance. We investigate the following core design choices:• The order of the previous instructions. We compare the following options: (1) from lowest tohighest (our default setting); (2) from highest to lowest; (3) random. Figures 7(a) and 7(b) showthat the default setting achieves better final accuracies and converges faster. One hypothesis isthat the optimizer LLM output is affected more by the past instructions closer to the end of themeta-prompt. This is consistent with the recency bias observed in Zhao et al. (2021), whichstates that LLMs are more likely to generate tokens similar to the end of the prompt.• The effect of instruction scores. In terms of how to present the accuracy scores, we compare threeoptions: (1) rounding the accuracies to integers, which is equivalent to bucketizing the accuracyscores to 100 buckets (our default setting); (2) bucketizing the accuracies to 20 buckets; (3)not showing the accuracies, only showing the instructions in the ascending order. Figures 7(c)and 7(d) show that the accuracy scores assists the optimizer LLM in better understanding thequality difference among previous instructions, and thus the optimizer LLM proposes better newinstructions that are similar to the best ones in the input optimization trajectory.• The effect of exemplars. We compare three options: (1) showing 3 exemplars from the task(default); (2) showing 10 exemplars from the task; (3) no exemplars

One challenge of prompt optimization is the sensitivity of model performance to subtle changes inthe instruction. For example, with the PaLM 2-L scorer on the GSM8K test set, “Let’s think stepby step.” achieves accuracy 71.8, “Let’s solve the problem together.” has accuracy 60.5, while theaccuracy of “Let’s work together to solve this problem step by step.” is only 49.4, although it is thesemantic combination of the two upper instructions. This behavior increases both the variance acrosssingle-step instructions and the oscillation during optimization, and motivates us to generate multipleinstructions at each step to improve the optimization st

We assess the transferability of found prompts to different datasets of the same domain, where weevaluate the top instructions found for GSM8K on two more math reasoning benchmarks Multi-Arith (Roy & Roth, 2016) and AQuA (Ling et al., 2017). Table 6 shows that our optimized promptsalso outperform baseline prompts with different scorer LLMs on these two benchmarks

Similar to GSM8K, we observe upward trends in optimization curves on almost all BBH tasks, asshown in Figure 6. See Figure 23 and 24 in Appendix D for more curves on other BBH tasks.We next show some examples of instructions found through the course of optimization. On the taskruin_names, starting from the empty instruction (with 64.0 training accuracy), with the text-bisonscorer and the PaLM 2-L-IT optimizer, the following instructions are generated:• “Consider the following when editing artist or movie names humorously:” at Step 1 with trainingaccuracy 72.0;• “When making humorous edits of artist or movie names, you can change one or more letters oreven create puns by adding new words that sound similar.” at Step 18 with training accuracy80.0;• “We can make humorous edits of artist/movie names by changing letters to create new wordsthat are similar in sound but have different meanings. For example, The Police can be changedto The Polite, The Abyss can be changed to Toe Abyss, and Schindler’s List can be changed toSchindler’s Lost.” at Step 38 with training accuracy 82.0.Although the above instructions are semantically similar, a paraphrase by the optimizer LLM offers anotable accuracy improvement. We further highlight this observation in Section 5.2.3.

Figure 5 visualizes the per-task accuracy difference on all 23 BBH tasks compared to the instruction“Let’s think step by step.” (Kojima et al., 2022) and the empty instruction, and we present the concreteaccuracies in Table 7 of Appendix E. We show that the instructions found by OPRO outperform“Let’s think step by step.” on almost all tasks by a large margin: our instructions outperform by over5% on 19/23 tasks with the PaLM 2-L scorer, and on 15/23 tasks with the text-bison scorer.Our prompt optimization algorithm also improves instructions from the empty starting point by over5% on most tasks: 20/23 with the PaLM 2-L scorer and 15/23 with the text-bison scorer

For each task, we utilize a subset of 20% examples for prompt optimization, andthe rest examples are for testing. We show experimental results on more variants of the instructionposition and initialization in Appendix

Table 4 summarizes top instructions found on GSM8K with different scorer and optimizer LLMs.We observe that:• The styles of instructions found by different optimizer LLMs vary a lot: PaLM 2-L-IT andtext-bison ones are concise, while GPT ones are long and detailed.• Although some top instructions contain the “step-by-step” phrase, most others achieve a compa-rable or better accuracy with different semantic meanings.10

Different from other optimizer LLMs that are instruction-tuned, the pre-trained PaLM 2-L performs better when the prompt is formatted in a few-shot manner.Therefore, we include two initial instructions to start the optimization: the empty instruction (witha training accuracy 32.2) and “The answer is” (with a training accuracy 33.3).

te that although our default setting is to run OPRO for 200 steps in prompt optimization, weneed much fewer steps if the goal is to find some outstanding instructions. An example is that theFigure 1(a) experiment found “Let’s do the math!” at Step 6 with training accuracy 78.2, almostmatching the “Take a deep breath and work on this problem step-by-step.” found at the 107th stepwith training accuracy 80.2, at a point where the optimization curve is still trending upwards. This isbecause a leap in our optimization curve does not always correspond to a much better instruction beingdiscovered; instead, it can be due to a large qualitative improvement of all 8 generated instructions inthis step. The latter usually happens several steps after the former: after a much better instruction isdiscovered in one step, the meta-prompt gradually gets rid of worse instructions in the latter steps bygenerating instructions similar to the much-better one. The top instructions kept in the meta-promptgradually improves in this procedure. At a point when the meta-prompt only triggers higher qualityinstructions, the leap happens

The optimization curves also generally show a decrease of the variance among the accuracies ofinstructions generated at each step, indicating that the optimizer LLM generates distributionallybetter instructions throughout the optimization.

We observe that the optimization curveshows an overall upward trend with several leaps throughout the optimization process

Section 1 shows prompt optimization curves with pre-trained PaLM 2-L as scorerand PaLM 2-L-IT as optimizer, and the initial instruction is “Let’s solve the problem” with a(approximated, and same below) training accuracy of 60.5.

For prompt optimization, we randomly sample 3.5% examples from the GSM8K training set. Thesame subset is used throughout optimization, so that the task accuracies computed at intermediateoptimization steps are approximations of the training accuracy on all 7,473 training examples. Thisbalances the evaluation cost with the generalization performance. After the optimization procedurefinishes, we evaluate the found instructions on the entire GSM8K test set.

We would like to note that OPRO is designed for neither outperforming the state-of-the-art gradient-based optimization algorithms for continuous mathematical optimization, norsurpassing the performance of specialized solvers for classical combinatorial optimization problemssuch as TSP. Instead, the goal is to demonstrate that LLMs are able to optimize different kindsof objective functions simply through prompting, and reach the global optimum for some small-scale problems. Our evaluation reveals several limitations of OPRO for mathematical optimization.Specifically, the length limit of the LLM context window makes it hard to fit large-scale optimizationproblem descriptions in the prompt, e.g., linear regression with high-dimensional data, and travelingsalesman problems with a large set of nodes to visit. In addition, the optimization landscape of someobjective functions are too bumpy for the LLM to propose a correct descending direction, causing theoptimization to get stuck halfway. We further elaborate our observed failure cases in Appendix A.

On the other hand, the performance of OPRO degrades dramatically on problems with larger sizes.When n = 10, all LLMs find the optimal solutions for every evaluated problem; as the problem sizegets larger, the OPRO optimality gaps increase quickly, and the farthest insertion heuristic starts tooutperform all LLMs in the optimality gap.

We present the results in Table 3. We randomly generate 5 problem instances for each number ofnodes n. In addition to measuring the optimality gap, on problems where the LLM finds the optimalsolutions, we also show the number of optimization steps taken to reach the global optimum. First,we observe that gpt-4 significantly outperforms gpt-3.5-turbo and text-bison across allproblem sizes. Specifically, on smaller-scale problems, gpt-4 reaches the global optimum about 4×faster than other LLMs. On larger-scale problems, especially with n = 50, gpt-4 still finds solutionswith a comparable quality to heuristic algorithms, while both text-bison and gpt-3.5-turboget stuck at local optima with up to 20× worse optimality gaps.

Nearest Neighbor (NN). Starting from an initial node, the solution is constructed withthe nearest neighbor heuristic: At each step, among the remaining nodes that are not included inthe current partial solution, NN selects the node with the shortest distance to the end node of thepartial solution, and adds it as the new end node. The process finishes when all nodes have beenadded to the solution.• Farthest Insertion (FI). One caveat of the nearest neighbor heuristic is that it doesnot take the distance between the start and end node into consideration when constructing partialsolutions. To address this issue, FI aims to optimize the cost of inserting new nodes into thepartial solution at each step. Define the minimal insertion cost of adding a new node k as

We generate the problem instances by sampling n nodes with both x and y coordinates in [−100, 100].We use the Gurobi solver (Optimization et al., 2020) to construct the oracle solutions and compute theoptimality gap for all approaches, where the optimality gap is defined as the difference between thedistance in the solution constructed by the evaluated approach and the distance achieved by the oraclesolution, divided by the distance of the oracle solution. Besides evaluating OPRO with differentLLMs including text-bison, gpt-3.5-turbo and gpt-4, we also compare OPRO to thefollowing heuristics

consider the Traveling Salesman Problem (TSP) (Jünger et al., 1995; Gutin & Punnen, 2006),a classical combinatorial optimization problem with numerous algorithms proposed in literature,including heuristic algorithms and solvers (Rosenkrantz et al., 1977; Golden et al., 1980; Optimizationet al., 2020; Applegate et al., 2006; Helsgaun, 2017), and approaches based on training deep neuralnetworks (Kool et al., 2019; Deudon et al., 2018; Chen & Tian, 2019; Nazari et al., 2018). Specifically,given a set of n nodes with their coordinates, the TSP task is to find the shortest route that traversesall nodes from the starting node and finally returns to the starting node.

The number of unique (w, b) pairs explored by each model is fewer than exhaustive search,indicating these models are able to to do black-box optimization: compare the numbers andpropose a descent direction.• The text-bison and gpt-4 models outperform gpt-3.5-turbo in convergence speed:they arrive at the optima with fewer steps. The gpt-4 model also outperforms in finding theoptima with fewer explored unique points. Taking a closer look at the optimization trajectory, wesee gpt-4 is the best at proposing a reasonable next step from the history: for example, whenthe history shows the objective values of (w, b) = (8, 7), (w, b) = (8, 6), and (w, b) = (8, 5)are decreasing, it has a highest chance to propose (w, b) = (8, 4) for evaluation.• The problem becomes harder for all models when the ground truth moves farther from thestarting region: all models need more explorations and more steps

Both w and b startfrom 5 random starting points in [10, 20]. We use temperature 1.0 for all models. We run each setting5 times. The starting points are the same across optimizer LLMs but are different across 5 runs, andare grouped by: within the starting region, outside and close to the starting region, and outside andfarther from the starting region. Bold numbers indicate the best among three LLMs in each setting

We study three settings of wtrue and btrue: within the startingregion [10, 20] × [10, 20], “near outside” (each of wtrue and btrue is outside the starting region but thedistance is less than 10), and “far outside” (each of wtrue and btrue is outside the starting region andthe distance is greater than 10). We see:

Optimization problem examples. The problem description includes a few examples taken from thetraining set to demonstrate the task for the generated instructions. For example, from the input-outputpair in Figure 3, we can infer this is a math word problem. The input-output pair also demonstratesthe position where the generated instruction will be added to, and this is essential for the optimizerLLM to generate instructions of the same style. In each optimization step, we add several (three forexample) training examples to the meta-prompt by random sampling the training set or choose theones the previous instructions fall short of.Optimization trajectory. The optimization trajectory includes instructions generated from the pastoptimization steps, along with their scores. The old instructions and scores are sorted by the score inascending order. The score is the training accuracy in prompt optimization. We only keep instructionswith the highest scores in the meta-prompt in consideration of the LLM context length limit.Meta-instructions. We also add meta-instructions: the instructions to the optimizer LLM that explainthe optimization goal and instruct the model how to use the above information. The meta-instructionsmay also specify the desired generated instruction format for easier parsing.

Our primary evaluation benchmarks are GSM8K (Cobbe et al., 2021) and Big-BenchHard (BBH) (Suzgun et al., 2022). GSM8K is a benchmark of grade school math word problemswith 7,473 training samples and 1,319 test samples, where chain-of-thought prompting (Wei et al.,2022) and the zero-shot instruction “Let’s think step by step.” (Kojima et al., 2022) have drasticallyimproved the performance over the standard prompting. BBH is a suite of 23 challenging BIG-Benchtasks (Srivastava et al., 2022) that covers a wide range of topics beyond arithmetic reasoning, includingsymbolic manipulation and commonsense reasoning. Each task contains up to 250 examples in total.To examine the transferability of the optimized instructions, we also evaluate the instructions op-timized for GSM8K on two other mathematical reasoning datasets, i.e., MultiArith (Roy & Roth,2016) and AQuA (Ling et al., 2017)

Models. The LLMs we use as the optimizer and the scorer are:• Optimizer LLM: Pre-trained PaLM 2-L (Anil et al., 2023), instruction-tuned PaLM 2-L(denoted PaLM 2-L-IT), text-bison, gpt-3.5-turbo, and gpt-4.• Scorer LLM: Pre-trained PaLM 2-L and text-bison.With pre-trained PaLM 2-L as the scorer, the optimizer LLM generates A_begin instructions.Since text-bison has been instruction-tuned, the optimizer LLM generates Q_begin and Q_endinstructions when text-bison is used as the scorer.

n example of the meta-prompt for prompt optimization with instruction-tuned PaLM 2-L(PaLM 2-L-IT) on GSM8K, where the generated instruction will be prepended to the beginningof “A:” in the scorer LLM output (A_begin in Section 4.1). <INS> denotes the position where thegenerated instruction will be added. The blue text contains solution-score pairs; the purple textdescribes the optimization task and output format; the orange text are meta-instructions

Q_begin: the instruction is added before the original question.• Q_end: the instruction is added after the original question.• A_begin: the instruction is added to the beginning of the scorer LLM output. This is applicableto pretrained LLMs without instruction tuning, where the prompt is formatted as a sequence ofQA pairs.

We denote the LLM for objective function evaluation as the scorer LLM, and the LLMfor optimization as the optimizer LLM.The output of the optimizer LLM is an instruction, which is concatenated to the question part of everyexemplar and prompts the scorer LLM. We consider the following positions to insert the instruction:

We prompt the meta-prompt 8times to generate at most 8 new (w, b) pairs in each step to improve optimization stability. Then weevaluate the objective value of the proposed pair and add it to history. We do black-box optimization:the analytic form does not appear in the meta-prompt text. This is because the LLM can oftencalculate the solution directly from the analytic form

In each step, we prompt an instruction-tuned LLM with a meta-prompt that includes the best 20 (w, b) pairs in history and their sortedobjective values. The meta-prompt then asks for a new (w, b) pair that further decreases the objectivevalue

We study the setting in which the independentand dependent variables X and y are both one-dimensional and an intercept b is present, so thatthere are two one-dimensional variables w, b to optimize over. In a synthetic setting, we sampleground truth values for one-dimensional variables wtrue and btrue, and generate 50 data points byy = wtruex + btrue + ε, in which x ranges from 1 to 50 and ε is the standard Gaussian noise.

t the solution generation step, the LLM generates new solutions with the meta-prompt as input. Thefollowing are the key optimization challenges we address in this stage.Optimization stability. In the optimization process, not all solutions achieve high scores andmonotonically improve over prior ones. Due to the sensitivity of in-context learning to the prompt,LLM output can be drastically affected by low-quality solutions in the input optimization trajectory,especially at the beginning when the solution space has not been adequately explored. This sometimesresults in optimization instability and large variance. To improve stability, we prompt the LLM togenerate multiple solutions at each optimization step, allowing the LLM to simultaneously exploremultiple possibilities and quickly discover promising directions to move forward.Exploration-exploitation trade-off. We tune the LLM sampling temperature to balance betweenexploration and exploitation. A lower temperature encourages the LLM to exploit the solution spacearound the previously found solutions and make small adaptations, while a high temperature allowsthe LLM to more aggressively explore solutions that can be notably differen

Optimization trajectory. Besides understanding natural language instructions, LLMs are alsoshown to be able to recognize patterns from in-context demonstrations (Wei et al., 2023; Madaan &Yazdanbakhsh, 2022; Mirchandani et al., 2023). Our meta-prompt makes use of this property and in-structs the LLM to leverage the optimization trajectory for generating new solutions. Specifically, theoptimization trajectory includes past solutions and their optimization scores, sorted in the ascendingorder. Including optimization trajectory in the meta-prompt allows the LLM to identify similarities ofsolutions with high scores, encouraging the LLM to build upon existing good solutions to constructpotentially better ones without the need of explicitly defining how the solution should be updated.

Optimization problem description. The first part is the text description of the optimization problem,including the objective function and solution constraints. For example, for prompt optimization,the LLM can be instructed to “generate a new instruction that achieves a higher accuracy”, and wedenote such instructions in the meta-prompt as meta-instructions. We can also provide customized

illustrates the overall framework of OPRO. In each optimization step, the LLM generatescandidate solutions to the optimization task based on the optimization problem description andpreviously evaluated solutions in the meta-prompt. Then the new solutions are evaluated and added tothe meta-prompt for the subsequent optimization process. The optimization process terminates whenthe LLM is unable to propose new solutions with better optimization scores, or a maximum numberof optimization steps has reached

eachoptimization step in our work generates new prompts that aim to increase the test accuracy based ona trajectory of previously generated prompts, instead of editing one input prompt according to naturallanguage feedback (Pryzant et al., 2023) or requiring the new prompt to follow the same semanticmeaning (Zhou et al., 2022b)

The meta-prompt contains two core pieces of information. The first piece ispreviously generated prompts with their corresponding training accuracies. The second piece is theoptimization problem description, which includes several exemplars randomly selected from thetraining set to exemplify the task of interest

wever, the large and discrete prompt space makes itchallenging for optimization, especially when only API access to the LLM is available. Followingprior work on continuous and discrete prompt optimization (Lester et al., 2021; Li & Liang, 2021;Zhou et al., 2022b; Pryzant et al., 2023), we assume a training set is available to compute the trainingaccuracy as the objective value for optimizatio

the optimal prompt formats can be model-specific and task-specific (Ma et al., 2023;Chen et al., 2023c). Therefore, prompt engineering is often important for LLMs to achieve goodperformance (Reynolds & McDonell, 2021).

Specifically, we focus on natural language tasks where both the taskinput and output are texts. LLMs are shown to be sensitive to the prompt format

Their ability to understand natural language lays out a new possibility for optimization: instead offormally defining the optimization problem and deriving the update step with a programmed solver,we describe the optimization problem in natural language, then instruct the LLM to iteratively generatenew solutions based on the problem description and the previously found solutions. Optimizationwith LLMs enables quick adaptation to different tasks by changing the problem description in theprompt, and the optimization process can be customized by adding instructions to specify the desiredproperties of the solutions.

We first showcaseOPRO on linear regression and traveling salesman problems, then move on to ourmain application in prompt optimization, where the goal is to find instructionsthat maximize the task accuracy

In this work, we propose Optimization by PROmpting(OPRO), a simple and effective approach to leverage large language models (LLMs)as optimizers, where the optimization task is described in natural language. Ineach optimization step, the LLM generates new solutions from the prompt thatcontains previously generated solutions with their values

In the decoding phase, we employ a greedy spansection (Zaratiana et al., 2022) that selects en-tity spans based on matching scores, to ensuretask/dataset specific constraints. This strategy is ap-plied independently to each sentence. Only, spans(i, j) with matching scores φ(i, j, c) > 0.5 are con-sidered for selection.Flat NER: The algorithm chooses the highest-scoring non-overlapping span and continues thisprocess until all spans are evaluated.Nested NER: Similar to Flat NER, but the algo-rithm allows selection of fully nested spans withinother entities while still avoiding partial overlaps.Algorithm Efficiency: The decoding is imple-mented using a priority queue for spans, ensuringan O(n log n) complexity, with n being the numberof candidate spans.

our objective is to optimize modelparameters to enhance the matching score for cor-rect span-type pairs (positive pairs) and reduce itfor incorrect pairs (negative pairs). A span (i, j)paired with an entity type t forms a positive pair(s ∈ P) if the span is labeled with type t in the train-ing data. Otherwise, it is a negative pair (s ∈ N ).The training loss for an individual example, com-prising spans S and entity types T , is defined as:LBCE = − ∑s∈S×TIs∈P log φ(s)+Is∈N log (1 − φ(s))(3)The variable s represents a pair of span/entitytype and I is an indicator function, which returns1 when the specified condition is true and 0 oth-erwise. This loss function corresponds to binarycross-entropy

To evaluatewhether a span (i, j) corresponds to entity type t,we calculate the following matching score:φ(i, j, t) = σ(STij qt) ∈ R (2)In this equation, σ denotes a sigmoid activationfunction. As we train with binary cross-entropyloss (see next sec. 2.2), φ(i, j, t) can be interpretedas the probability of the span (i, j) being of type t.

The computation of all span represen-tations can be easily parallelized. Moreover, we setan upper bound to the length (K=12) of the span inorder to keep linear complexity, without harmingrecall.

he entity representa-tion is computed by refining the initial represen-tation p using a two-layer feedforward network,resulting in q = {qi}M −10 ∈ RM ×D. The repre-sentation of a span starting at position i and endingat position j in the input text, Sij ∈ RD, is com-puted as:Sij = FFN(hi ⊗ hj ) (1)Here, FFN denotes a two-layer feedforward net-work, and ⊗ represents the concatenation operation.

h = {hi}N −10 ∈ RN ×D denotes the representationof each word in the input text. For words tokenizedinto multiple subwords, we use the representationof the first subword, which is a standard choice inthe NER literature

Let p = {pi}M −10 ∈ RM ×D represent the en-coder’s output for each entity type, correspondingto all the [ENT] token representation

put format The input to our model comprises aunified sequence combining entity types (expressedin natural language) and the input text from whichentities are to be extracted. The input format is asfollows:Human: What describes organization in the text ?Assistant: [‘Mcgill University’]Bidirectional LMs(BERT, DeBERTa)(0,1, person)(4,5, organization)[ENT] person [ENT] location [ENT] organization [SEP]+Alain Farley works at McGill Universitya) UniNER (prev) : Prompting LLM for Open NER.b) GLiNER (Ours): Prompting BiLM for Open NER.[ENT] [ENT] ... [ENT] [SEP] ...t0 t1 tM−1 x0 x2 xN−1[ENT] token represents a special token placedbefore each entity type and the [SEP] token func-tions as a delimiter, separating the sequence of en-tity types from the input text. They are initializedrandomly at the start of training

Our model has threemain components: i) a pre-trained textual encoder(a BiLM such as BERT), ii) a span representationmodule which computes span embeddings from to-ken embeddings, iii) an entity representation mod-

turally solves the scalabilityissues of autoregressive models and allows for bidi-rectional context processing, which enables richerrepresentations

n our work, we propose a model that addressesthe above-mentioned problems. Instead of relyingon large autoregressive models, we utilize smaller-scale Bidirectional Language Models (BiLM), suchas BERT (Devlin et al., 2019) or deBERTa (Heet al., 2021). The core concept of our model in-volves treating the task of Open NER as matchingentity type embeddings to textual span represen-tations in latent spac

While these works have achievedremarkable results, they present certain limitationswe seek to address: They use autoregressive lan-guage models, which can be slow due to token-by-token generation; Moreover, they employ largemodels with several billion parameters, limitingtheir deployment in compute-limited scenarios.Furthurmore, as NER is treated as a text gener-ation problem, the generation of entities is donein several decoding steps, and there is no way toperform the prediction of multiple entity types inparalle

powerfulLLMs typically consist of billions of parametersand thus require substantial computing resources.Although it is possible to access some LLMs viaAPIs (OpenAI, 2023), using them at scale can incurhigh cost

Traditional NER models arelimited to a predefined set of entity types. Expand-ing the number of entity types can be beneficial formany applications but may involve labeling addi-tional datasets

veraging a bidirectional transformer en-coder, our model, GLiNER, facilitates parallelentity extraction, an advantage over the slowsequential token generation of LLMs. Throughcomprehensive testing, GLiNER demonstratestrong performance, outperforming both Chat-GPT and fine-tuned LLMs in zero-shot evalua-tions on various NER benchmarks

Data construction prompt. Fig. 6 shows theprompt used for Chinese distillation data construc-tion. We follow Zhou et al. (2024) to design theprompt for Chinese data construction. We adoptthe data construction prompt of Pile-NER-type 3,since it shows the best performance as in (Zhouet al., 2024).Figure 6: Data construction prompt for Chinese opendomain NER.Data processing. Following (Zhou et al., 2024),we chunk the passages sampled from the Sky cor-pus4 to texts of a max length of 256 tokens andrandomly sample 50K passages. Due to limitedcomputation resources, we sample the first twentyfiles in Sky corpus for data construction, since thesize of the entire Sky corpus is beyond the pro-cessing capability of our machines. We conductthe same data processing procedures including out-put filtering and negative sampling as in UniNER.Specifically, the negative sampling strategy for en-tity types, is applied with a probability proportionalto the frequency of entity types in the entire con

ference with out-domain examples. Duringinference, since examples from the automaticallyconstructed data is not aligned with the domainsand schemas of the human-annotated benchmarks,we refer to them as out-domain examples. Fig. 4shows the results of inference with out-domain ex-amples using diverse retrieval strategies. We usethe model trained with NN strategy here. After ap-plying example filtering such as BM25 scoring, in-ference with out-domain examples shows improve-ments compared to the baseline, suggesting theneed of example filtering when implementing RAGwith out-domain examples

Training with diverse retrieval strategies. Fig.3 visualize the results of training with various re-trieval strategies. We conduct inference with andwithout examples for each strategy, and set the re-trieval strategy of inference the same as of training.The most straight forward method NN shows bestperformances, suggesting the benefits of semanti-cally similar examples. Random strategy, though in-Figure 4: Impacts of inferece with out-domain examplesusing various retrieval strategies. The average F1 valueof the evaluated benchmarks are reported. w/o exmp.means inference without example. Applying examplefiltering strategy such as BM25 filtering benefits RAGwith out-domain examples.Figure 5: Impacts of inference with in-domain examples.The average F1 value of the evaluated benchmarks arereported. N -exmp. means the example pool of size N .Sufficient in-domain examples are helpful for RAG.ferior to NN, also shows improvements, indicatingthat random examples might introduce some gen-eral information of NER taks to the model. Mean-while, inference with examples does not guaranteeimprovements and often hurt performances. Thismay due to the differences of the annotation schemabetween the automatically constructed data and thehuman-annotated benchmarks

Diverse retrieval strategies. The followingstrategies are explored in the subsequent analysis.(1) Nearest neighbor (NN), the strategy used in themain experiments, retrieves k nearest neighborsof the current sample. (2) Nearest neighbor withBM25 filter (NN, BM), where we apply BM25 scor-ing to filters out NN examples not passing a prede-fined threshold. Samples with no satisfied exam-ples are used with the vanilla instruction template.(3) Diverse nearest neighbor (DNN), retrieves Knearest neighbors with K >> k and randomly se-lects k examples from them. (4) Diverse nearestwith BM25 filter (DNN,BM), filters out DNN exam-ples not reaching the BM25 threshold. (5) Random,uniformly selects k random examples. (6) Mixednearest neighbors (MixedNN), mixes the using ofthe NN and random retrieval strategies with theratio of NN set to a.

We explore the impacts of diverse retrieval strate-gies. We conduct analysis on 5K data size for costsaving as the effect of RA-IT is consistent acrossvarious data sizes as shown in Section 3.4. Wereport the average results of the evaluated bench-marks here

The main results are summarized in Table 1 and2 respectively. We report the results of inferencewithout examples for RA-IT here, since we foundthis setting exhibits more consistent improvements.The impacts of inference with examples are studiedin Section 3.5. As shown in the tables, RA-ITshows consistent improvements on English andChinese across various data sizes. This presumablybecause the retrieved context enhance the model

We conduct a preliminary study on IT data effi-ciency in targeted distillation for open NER byexploring the impact of varous datas sizes: [0.5K,1K, 5K, 10K, 20K, 30K, 40K, 50K]. We use vanillaIT for preliminary study. Results are visualized inFig. 2. The following observations are consistentin English and Chinese: (1) a small data size al-ready surpass ChatGPT’s performances. (2) Perfor-mances are improving as the data sizes increased to10K or 20K, but begin to decline and then remainat a certain level as data sizes further increased to50K. Recent work for IT data selection, Xia et al.Figure 2: Preliminary study of IT data efficiency foropen NER in English (left) and Chinese (right) scenar-ios, where the training data are Pile-NER and Sky-NERrespectively. Average zero-shot results of evaluatedbenchmarks are illustrated. The performance does notnecessarily improve as the data increases.(2024); Ge et al. (2024); Du et al. (2023) also findthe superior performances of only limited data size.We leave selecting more beneficial IT data for IEas future work. Accordingly, we conduct mainexperiments on 5K, 10K and 50K data sizes

Training data: For English, we use thetraining data Pile-NER released by Zhou et al.(2024). For Chinese, we use the training data Sky-NER constructed in this paper as described in Sec-tion 3.2. We use LoRA (Hu et al., 2021) to trainmodels. Retrieval: We adopt GTE-large2 (Liet al., 2023) to generate text embeddings and setk = 2 in main experiments. Evaluation: Wemainly focus on the zero-shot evaluation. ForEnglish, we adopt benchmarks CrossNER, MIT-Movie and MIT-restaurant following Zhou et al.(2024). For Chinese, we collect eight benchmarksacross diverse domains, of which details are in Ap-pendix D. We report micro-F1 value

Retriever. We use sentence embedding-based re-trieval and adopt cosine similarity as our similaritymetric. We retrieve the k nearest neighbors as con-text. We also investigate various retrieval strategiesfor both training and inference stages

RA-IT. We explore an alternative way to conductIT in targeted distillation: we introduce RA-IT, acontext-enhanced tuning approach, of which theoverview is in Fig. 1. In our RA-IT approach,each data is augmented with a retrieved context,which consists of k semantically similar exam-ples retrieved from the training dataset. The re-trieved context is prepended to the original conver-sation, forming the retrieval augmented instruction.By fine tuning LMs in this recipe, we equip theLMs with the ability to generate NER answer withon-demand RAG. This means we could flexiblyadapting LMs to different scenarios by determin-ing whether to use RAG during inference based onthe specific characteristics of the scenario.

Vanilla IT. The original instruction tuning tem-plate used in targeted distillation is shown in thebottom part of Fig. 1, which we refer to as VanillaIT, where each passage and its associated entityoutput are converted into a multi-turn conversation.

reliminary: Targeted Distillation. We followUniNER (Zhou et al., 2024) to conduct our studyin the setting of targeted distillation, where theysuccessfully distill the strong capability of Chat-GPT in open NER into smaller models, without anyhuman-annotated data. The pipeline is as follows:(1) Data construction. They sample inputs froma large corpus across diverse domains, then useChatGPT to automatically generate NER outputs.(2) Distillation. After obtaining the automaticallyconstructed data, they apply IT to distill the openNER capability of ChatGPT into smaller models

1) Weempirically study the RA-IT framework for openNER. We prepare the retrieval augmented instruc-tion data with semantically similar examples. Weconduct thorough experimental analysis to studythe impact of various retrieval strategies. (2) We

(1) RA-ITachieves consistent improvements on various datasizes, suggesting the need for context-enhancedfine-tuning. (2) Retrieving semantically similar ex-amples benefits the most for training among variousretrieval strategies. Random retrieval also exhibitsimprovement but shows inferior performance tosimilar examples. (3) Retrieving out-domain ex-amples for inference requires applying examplefiltering strategies to achieve improvements. Pro-viding in-domain examples benefits inference.

our RA-IT approach, for each training sample,we retrieve semantically similar examples from thetraining dataset and prepend them to the original in-struction, forming the context-enhanced instruction.We also explore the impacts of diverse retrievalstrategies. Moreover, we construct a Chinese ITdataset for open NER and evaluate our methodin both English and Chinese scenarios. We con-duct thorough experiments across various data sizesand obtain the following key finding

The previous work UniNER (Zhou et al., 2024)distills the strong capability of ChatGPT in openNER into smaller models through IT without anyhuman-annotated data. We follow this line andinvestigate RA-IT under this targeted distillationsetting. Other works of IT for IE like Sainz et al.(2024); Li et al. (2024) using code-style instructiondata, are orthogonal to this work since RA-IT canbe integrated into various instruction styles.

Inthis paper, we explore Retrieval AugmentedInstruction Tuning (RA-IT) for IE, focusingon the task of open named entity recognition(NER). Specifically, for each training sample,we retrieve semantically similar examples fromthe training dataset as the context and prependthem to the input of the original instruction.

n summary, our contributions are three-fold: (i)We propose a framework CLUSTERLLM that uti-lizes sentence relations predicted from API-basedLLMs to guide clustering. Furthermore, it allowsusers to provide textual instructions and/or few-shot annotations to specify preferences on cluster-ing. (ii) In order to reduce API-queries, we proposea novel entropy-based sampling strategy to find themost informative triplets. Additionally, we utilizepairwise data sampled from hierarchical cluster-ing to determine cluster granularity. (iii) Extensiveexperiments show that our proposed method canimprove clustering performance at ∼$0.2 for per-spective and ∼$0.4 for granularity with GPT-3.5.

n Stage 2, we first obtain the cluster hierarchythat starts from instance-level clusters and itera-tively merge two closest clusters until the entiredataset. And then we prompt LLMs to determinecluster granularity with a few annotated data pairsas demonstrations. We construct the data pairsto prompt by sampling from two clusters that aremerged at each step of hierarchical clustering, sothat they cover a wide range of granularities. Andthe final decision is made by measuring consistencybetween each level of clustering and predictions.

In Stage 1, we prompt LLMs with a triplettask that predicts which one of the two candidatechoices is closer to anchor instance to understandthe user-preferred perspectives. We choose thistriplet task because (a) it is irrelevant with clustergranularity and (b) the produced triplets can fine-tune small embedder towards the right perspective.In order to improve sample efficiency, we furtherpropose entropy-based triplet sampling to find themost informative triplets. Specifically, we first cal-culate entropy for each instance based on clusterassignment probabilities, and then identify thosewith highest entropy. Two candidate choices arethen sampled from its nearest clusters to guaranteethey are close enough to the ancho

We propose CLUSTERLLM, a framework thatutilizes LLM to guide a small embedder for findingtext clusters with a low cost, as shown in Figure 1.It comprises two stages that are specially designedfor two aspects of clustering: (1) perspective, i.e.,the grouping criterion such as topic, intent and emo-tion and (2) granularity, i.e. the scope of clusters

n this paper, we provide insights on the ques-tion: Can we leverage API-based LLMs to guidetext clustering efficiently? We attack this challeng-ing question by drawing inspiration from an obser-vation that humans represent an instance throughcomparing with others

State-of-the-artlarge language models (LLMs) such as recent GPTseries (Brown et al., 2020; Ouyang et al., 2022;OpenAI, 2023) have demonstrated extraordinarylanguage capabilities for various NLP applicationshowever, these GPT models can only be utilizedthrough the APIs without accessible embeddingvectors for clustering. Hence, LLMs cannot bedirectly applied on text clustering tasks

ext clustering, as a fundamental task in natural lan-guage processing (NLP), has a wide spectrum ofapplications, such as identifying public perceptionfrom social media (Park et al., 2022), analysingcause of accidents (Xu et al., 2022), and detectingemerging research topics (Martínez et al., 2022). Acommon practice for text clustering is to apply clus-tering algorithms (MacQueen, 1967; Zhang et al.,∗ Corresponding author.1The cost is calculated with gpt-3.5-turbo.Texts CABTraditional Text ClusteringChatGPT(API-based)🔒ClusterLLMCABTexts CABChatGPT(API-based)🔒A should be closerto C than B🧐Not Applicable ⛔Embedder(Instructor,E5,GTR ...)Embedder(Instructor,E5,GTR ...)Figure 1: LLMs like ChatGPT are not applicable for textclustering directly because of the inaccessible embed-dings. CLUSTERLLM resolves the dilemma by leverag-ing LLM as a guide on text clustering.2021a) on top of pre-trained embedders (Muen-nighoff et al., 2022; Wang et al., 2022; Su et al.,2022) which could achieve higher performancewith better pre-training quality

Due to the diversity of possibilities in human lan-guage, it is rare for the same idea to be expressedidentically in multiple documents unless one ex-pression is derived from the other, or both are quot-ing from a shared source. This observation moti-vates deduplicating exact substrings. We call ourapproach EXACTSUBSTR. When two examplesxi and xj share a sufficiently long substring (thatis, a substring for which xa..a+ki = xb..b+kj ), thatsubstring is removed from one of them. Basedon statistical analyses (§B), we select k = 50 to-kens as the minimum matching substring length.3

We introduce two complementary methodsfor performing deduplication. First, using a suf-fix array (Manber and Myers, 1993), we removeduplicate substrings from the dataset if they oc-cur verbatim in more than one example. Second,we use MinHash (Broder, 1997), an efficient algo-rithm for estimating the n-gram similarity betweenall pairs of examples in a corpus, to remove entireexamples from the dataset if they have high n-gramoverlap with any other example

n our research, we do not focus on the impact ofduplicate text in pretrained models on downstreambenchmark tasks; instead we address how duplicatetext in the LM training and validation sets impactsmodel perplexity and the extent to which generatedtext included memorized content

GPT-3(Brown et al., 2020, §5) did the reverse and re-moved downstream evaluation examples from theirtraining data by conservatively filtering out anytrain set examples with a 13-gram overlap withany evaluation example. Up to 90% of tasks wereflagged as potentially contaminate

Trinh and Le (2018, Appendix B) removeddocuments from their CommonCrawl-based trainset that overlapped substantially with the common-sense reasoning used for evaluatio

ontamination of downstream tasks. Whenmodels are trained on datasets constructed by crawl-ing the Internet, it is possible the model will trainon the test set of downstream target tasks

We propose two scalable techniques to detectand remove duplicated training data. Exact sub-string matching identifies verbatim strings that arerepeated. This allows us to identify cases whereonly part of a training example is duplicated (§4.1).Approximate full document matching uses hash-based techniques (Broder, 1997) to identify pairsof documents with high n-gram overlap (§4.2).

We run all instruction tuning experiments fromthe Hungarian pretrained model using 6 gigabytesof instruction tuning text data (2 billion tokens)and the same training settings. Each experimentis repeated 3 times with different random datasetsamples. For more details on instruction tuningdatasets or training settings see appendix C.2, C.4and D.2

Given the same total amount of training data, we tested varying the percentage of English data (50%,25% and 0%) in the English/Hungarian bilingual data mixture. All training is run for 30k steps. Wealso compare this to training a pure Hungarian model using only Hungarian data [31], a Hungariantokenizer, from scratch for 100k steps. All the training details can be found in appendix D.1.

We categorize all evaluation tasks into 4 categories. Multiple Choice, for this category we appendeach candidate answer to the prompt and pick the highest probability answer. Open-ended QuestionAnswering, where we let the model generate an answer for each question, and report the averageF1 score between the model output and the ground truth. Summarization, where we let the modelgenerate a summary and report the average ROUGE-2 score between the model output and groundtruth. Translation, where we let the model generate translated text and report the BLEU scorebetween the model output and the ground truth. When we report the score for each category, it is theaveraged score of all the evaluation tasks that we classified into that category in appendix E.

Training is done in a two stage pipeline. The first stage is adaptive pretraining (PT) where a basepretrained English 13B GPT-2 model (B) is continuously trained on a mixture composed of the newlanguage and English. Then, the adapted checkpoint is instruction tuned (IT) on a collection ofprompt completion pairs from the new language and English. For more information see appendixB,C

Once the datasets are prepared for both languages, we shuffle them at sample level, so that everybatch contains text from both languages during training. Note that in our experiments, we do notmake any further transformations to either the model or the datasets, after the data is preparedon each side, so that our study is orthogonal and complementary to existing proposed methods[24, 27, 28, 9, 12] focusing on training paradigm studies.

To adapt an existing tokenizer to a new language, tokens from the low resource language can beadded to the existing tokenizer’s vocabulary to improve its fertility. Fertility is defined as the averagenumber of tokens per word [22], and details about how we calculated it can be found in appendixA.1. In our work, instead of extending the tokenizer’s vocabulary, we replace the least frequenttokens from it with tokens from the new language. This way, we keep the model capability the sameby controlling the vocabulary and embedding table size. In particular, we train a BPE tokenizeron the new language with vocabulary size k and check the number of overlapping tokens o withthe original tokenizer. Then we replace the least important k − o non-overlapping tokens from theoriginal tokenizer with the new ones. We also reinitialize the corresponding embeddings in themodel. For more details see appendix A.2

fertility

We adapt an English-centric model to Hungarian and Thai, and our evaluations show that adding newtokens and mixing training data from both languages can retain the model’s English capabilities inaddition to improving the models ability to learn the new language. Some contemporary worksexplore similar, but far less efficient methods of training LLMs on low resource languages. [30]builds an English-Arabic bilingual LLM, but they train it from scratch; while [29] builds one forEnglish-Portuguese, but it does not optimize the tokenizer or mix the training data

How to efficiently encode the new language? Byte Pair Encoding (BPE) [15] tokenizers are com-monly used in LLMs including GPT[16, 17], Llama [18, 19] and BLOOM [1, 2]. These tokenizersare able to encode text at the byte level so that they can generalize to characters that are outsideof their vocabulary; this means that any BPE tokenizer can be used for all languages. However,the BPE tokenizer has poor tokenization efficiency if it was not trained on a given language. Forexample, the original English-centric GPT2 tokenizer with a vocabulary size of 50k needs to use3.8 times more tokens to encode Thai compared to a smaller tokenizer with a vocabulary size of 5kthat is trained on Thai. This will inevitably cost us 3.8 times more compute in both training andinference. Furthermore, it has been shown that models with sub-optimal tokenizers can also haveworse evaluation results [20, 21]. In our work, we show how to improve tokenizer fertility[22] byreplacing the least frequent tokens in the base model with tokens from the new language.How to avoid catastrophic forgetting? Many works have shown that when continuing to train aLLM on data from a new domain, it undergoes catastrophic forgetting of the original domain it wastrained on [23], and similar issues appear when training on a new language [23, 9, 24, 2, 25, 10, 26].Different training paradigms including instruction-align[24], MAD-X [27], (IA)3 [28] are proposed

Multilingual large language models have become prevalent recently [1, 2, 3, 4, 5, 6], and haveshown strong cross lingual knowledge and capability transfer [7, 8, 9, 10, 11, 12, 13]. However,these multilingual models tend to perform poorly on low-resource languages. On top of this, trainingmodels for low-resource languages from scratch is also challenging due to a lack of training data andprohibitive computational requirements. These challenges, along with the prevalence open sourcedEnglish models creates an interesting opportunity to see how they can be adapted to new languagesquickly, without wasting resources by pretraining from scratch. While prior work [9, 10, 11, 14, 13]has studied this concept, there are two important questions that warrant further investigation

When agents have different targets in a task, especially when the targets are adversarial, the task can become much more complicated. An example of such a task is a zero-sum game, where the total reward is fixed, and any reward gained by one agent results in an equal loss for another agent. A specific example can be found in MPE that in scenarios like simple_push, agent ONE is trying to gain more reward by getting closer to its target location while agent TWO gains reward by pushing agent ONE away from the target location. Moreover, the competitive-like mode can also be not so pure competitive. It can incorporate some cooperative agents’ relationships. This type of work mode is referred to as mixed mode. A representative task of mixed mode is MAgent, where agents are divided into several groups. Agents in the same group need to attack the enemy group cooperatively.

Another mode is collaborative, where agents can access individual rewards. Under this mode, the agents tend to work together, but the target varies between different agents. Sometimes individual rewards may cause some potential interest conflict. Collaborative task mode has less restriction and richer reward information for wilder algorithms development: il is a good solution for collaborative tasks, as each agent has been allocated an individual reward for doing a standard RL. Centralized Critic is a more robust algorithm family for collaborative tasks as the improved critic help agent coordinate using global information. Value Decomposition-based methods are still applicable for collaborative tasks as we can integrate all the individual rewards received into one (only the agents act simultaneously). Cooperative mode can also be transformed to collaborative as we can copy the global reward to each agent and treat them as an individual reward

The Cooperative-like task mode is prevalent in scenarios where agents are rewarded only when the team achieves a shared goal. This mode is considered a strict form of cooperation, where each agent cannot access its individual reward. In Cooperative tasks, agents must have a robust credit assignment mechanism to decompose the global reward and update their policies accordingly.

The current state of research on multi-agent reinforcement learning (MARL) is facing challenges regarding the diversity of multi-agent tasks and the categorization of MARL algorithms. These characteristics make it difficult to conduct a fair comparison of different algorithms and raise a question for researchers: should algorithms be developed for a specific task (task first) or for general tasks (algorithm first). This difficulty stems from the nature of multi-agent tasks, as well as the various learning styles and knowledge-sharing strategies.

Despite the simple task setting, however, the game is still very challenging as one agent needs to coordinate with another agent to achieve the highest reward: the joint action with the highest reward is not a good option from the view of the first agent if it is not willing to cooperate with another agent. Two-step Game evaluates whether an agent has learned to cooperate by sacrificing its reward for a higher team reward.

The first option for evaluating a new idea in MARL involves using a matrix and grid world task.One such example is the Two-step Game. In this task, two agents act in turn to gain the highest team reward. The task is very straightforward: two agents in the task the observation is a short vector with a length four two actions (A&B) to choose from

In the context of Multi-Agent Reinforcement Learning (MARL), a dataset corresponds to a collection of scenarios that comprise a single multi-agent task. Multi-agent tasks are customizable on a variety of aspects, such as the number of agents, map size, reward function, and unit status. This section provides a brief overview of the categories of multi-agent tasks, ranging from the simplest matrix game to real-world applications.

On the other hand, value decomposition-based algorithms can only be applied to cooperative and collaborative scenarios. These algorithms use a value decomposition technique to decompose the value function into individual value functions, one for each agent. The agents then learn their own policies based on their individual value functions. Since value decomposition-based algorithms do not use a centralized critic, they cannot be applied to competitive scenarios where the agents’ objectives conflict.

Centralized critic-based algorithms are applicable to all types of multi-agent tasks, including cooperative, collaborative, competitive, and mixed. These algorithms use a centralized critic to approximate the state-action value function, which enables agents to learn a policy that considers the actions of other agents and the global state

CTDE strikes a balance between coordination learning and deployment cost, making it a popular framework in MARL. Since multi-agent tasks involve numerous agents, learning a policy that aligns with the group target requires incorporating extra information from other sources. Thus, centralized training is the preferred choice. However, after training, the delivery of centralized information is too costly during deployment, leading to delays in decision-making. Furthermore, centralized execution is insecure, as centralized information can be intercepted and manipulated during transmission.

The CTDE framework, which stands for Centralized Training & Decentralized Execution, is a widely used approach in multi-agent reinforcement learning (MARL). In this setting, agents are trained together in a centralized manner where they can access all available information, including the global state, other agents’ status, and rewards. However, during the execution stage, agents are forced to make decisions based on their local observations, without access to centralized information or communication.

In a Partially Observable Markov Decision Process (POMDP), the system states are unobservable and probabilistically mapped to observations. The agent’s access to the system state is limited, and taking the same action can result in different observations. The observation is, however, still dependent on the system state. Hence, the agent must learn or hold a belief about its observation and learn a policy that accounts for all possible states.

In Wang et al. (2023a), the authors proposed to train the demonstrationretriever model with combined objectives: (1) knowledge distillation from the trained reward modelwhich can capture the preferences of LLMs over the retrieved candidates (2) InfoNCE-based con-trastive loss to incorporate the in-batch negatives. More specifically, the resulting loss function is asfollows:Lcombined = αLcont + Ldistill

nfoNCE Loss Another widely adopted training procedure is contrastive learning using the In-foNCE loss (Rubin et al., 2022; Cheng et al., 2023; Luo et al., 2023). When positive and negative11

The Determinantal Point Process model (Alex Kulesz, 2012) defines a proba-bility distribution over all the combinations of candidate demonstrations, giving high probability tosubsets that contain relevant and diverse items (Levy et al., 2022). It models diversity by incorporatingcross-candidate similarity scores, and models similarity via a per-candidate relevance score,

Wang et al. (2023a) and Li et al. (2023b) instead proposed to iterate the retrievermodel multiple times. More specifically, the retriever trained in iteration i will be employed toretrieve a new set of candidates for the subsequent iteration i + 1. Such an iterative training approachallows progressively improving retriever quality by mining better positive and hard negative examplesat each iteration

Distillation by KL Divergence Ye et al. (2023a) claims that although the InfoNCE loss has beenfound effective in training demonstration retrievers and can learn which examples might be superiorto others, it has the same treatment for all negative examples and the predicted scores from LLMare not fully utilized.

s an alternative to train a demonstration retriever using positive and negativeexamples, Shi et al. (2022) proposed to train the retriever by directly distilling the LLM’s scoringfunction. More specifically, the retriever model is designed to produce ranking scores that matchthe usefulness of a demonstration to help with the LLM inference; this is done by minimizing theKL-divergence between the top K examples score distribution from scoring LLM and the rankingscore distribution produced by the retriever

In the list-wise ranking objective, retriever can benefit from the full ranking of the candidate set to makeaccurate predictions for the most relevant demonstrations. However, obtaining the full rankinglist and calculating the loss function on top of it might be very expensive and time-consuming.Additionally, the model is trained to discern the relative preferences between examples withoutexplicitly determining whether an example can serve as an absolute good demonstration

he list-wise ranking approach looks at a list of candidate documents fora given query and tries to capture the correct ordering for it. Li et al. (2023b) proposed to inject theranking signals into the retriever using an approach inspired by LambdaRank (Burges, 2010

approach to collecting training data for demonstration retriever is to directlymeasure the similarity between the labels of the candidate demonstrations and the label of the query,and use this similarity as a proxy of the importance of a demonstration (Hu et al., 2022; Poesia et al.,2021).

nce a score is obtained, a retriever can be trained that predicts these scores directly (Ye et al.,2023a). Alternatively, the candidate demonstrations can be ranked for each query based on theirscores

ased on LLMs Signals A popular approach to collecting training examples is to use the su-pervisory signals from LLMs. In this case, a typical paradigm is to first employ some filteringmechanisms (Cheng et al., 2023) or unsupervised retrievers (e.g. BM25 and SBERT) (Luo et al.,2023) as the initial retriever, this step can help limit the pool size for mining the right training data.Then a scoring LLM, which serves as a proxy for the inference LLM, is used to score each candidatedemonstration d. Here the score is defined as s(e) = p(a|d, q) which is the conditional probability ofoutput answer a given the input query q and demonstration d. Another approach is to train a smallerreward model that can provide more fine-grained supervision for dense retrievers. For example, Wanget al. (2023a) proposed to finetune a cross-encoder model serving as a teacher model for training theretriever

Researchers thus have started to explore learning-based methods to further push theboundaries. A typical objective when designing a good demonstration retriever is: if an LLM finds ademonstration useful when being used as an illustrative example, the retriever should be encouragedto rank the demonstration higher

Pretrained Dual Encoder In the context of demonstration retrieval where the goal is to identifyrelevant examples for a given query, the query is typically a question, while the examples maycontain additional information such as answers, chains of thoughts, supporting knowledge, or evenfollow different patterns. Therefore, transforming them into a uniform embedding space to calculaterelevance might not be the most effective approach. In this case, LLM retrieval architectures suchas Dual Encoder that are pretrained on retrieval or question-answering tasks can better grasp theintricate relationships between complex logical concepts and reasoning processes by employingdifferent semantic embeddings for queries and candidates (Li and Qiu, 2023b). In practice, traininga dual-encoder can be highly expensive as it typically requires a large training corpus. Fortunately,there are publicly available pretrained retrievers, although not specifically optimized for few-shotretrieval tasks, already demonstrating success in helping LLMs to learn from the selected examples.Luo et al. (2023) studied applying GTR (Ni et al., 2021) to select semantically similar examples asdemonstrations, and empirically proved that this approach brought in better performance gain thanrandom fewshots for both PaLM (Chowdhery et al., 2023) and FLAN (Chung et al., 2022) models.GTR is a T5-based dual encoder model that is pretrained on the CommunityQA (Abujabal et al.,2019) and finetuned on the MS Marco dataset (Nguyen et al., 2016). Moreover, Khattab et al. (2022)reported results for employing ColBERTv2 (Santhanam et al., 2021) as the retrieval module in theirDEMONSTRATE–SEARCH–PREDICT (DSP) framework for ICL. ColBERTv2 is a state-of-artretrieval model that adopts the late interaction architecture (Khattab and Zaharia, 2020) and is trainedon the MS Marco dataset. In the proposed framework, it is used to retrieve both (i) related knowledgeduring the search stage and (2) top k similar examples as demonstrations.

Shi et al. (2022)extends the use case to cross-lingual few-shot retrieval in the Text to-SQL semantic parsing task, andthey use mSBERT (Reimers and Gurevych, 2019b), mUSE (Yang et al., 2019) and mT5 (Xue et al.,2020) as the baseline models for comparison. Other widely used baseline models for demonstrationretrieval include E5base (Wang et al., 2022b), SimCSE (Gao et al., 2021b). Instead of relying on“word matches” as in BM25, these sentence embedding similarity approaches can better capturesemantic similarity (

Sentence Embedding Similarity In this approach, queries and documents are encoded to thesame dense embedding space using an off-the-shelf sentence embedding model, and then similarityscores (e.g. cosine similarity) are calculated to rank the most relevant documents for each query.A rich collection of sentence embedding methodologies exists in the literature.

Term-based Similarity BM25 (Robertson et al., 2009) is one of the most popular term-basedscoring methods due to its simplicity and effectiveness in producing relevant results. It takes intoaccount both term frequencies and document lengths. It has been empirically demonstrated in variousworks (Luo et al., 2023; Rubin et al., 2022; Agrawal et al., 2022; Ye et al., 2023a; Dalvi et al., 2022)that using BM25 to select similar examples as few-shots in ICL can help improve the performance ofmany LLM inference tasks. While BM25 has become a standard baseline model in the field, it is notwithout its limitations. Due to its sole reliance on term frequency and document length, this approachmay overlook crucial aspects such as semantic meaning and sentence structure, potentially leading toinaccuracies in certain instances. Another drawback is that BM25 lacks the capability for fine-tuningin downstream tasks, making it less competitive compared to neural models which can be fine-tunedand customized for specific downstream tasks.

Free Form Corpus Another approach to deal with the lack of human-annotated data for similartasks is create pseudo-demonstrations from unstructured text. Toward this goal, Lyu et al. (2022)utilized the Demix dataset (Gururangan et al., 2022), which is not tailored for any specific task. Togenerate pusedo-demonstrations, a retriever selects the top-k most relevant sentences from the dataset.Subsequently, arbitrary labels are attached to each sentence to form the examples. Li et al. (2022)propose a synthetic question answering generation method to create QA pairs using the syntheticgenerated passages by an LLM

Unlabelled Queries with Automatically Generated Answers The previous three corpora allpresuppose the availability of human-annotated data. However, this assumption may not hold inreal-life scenarios, particularly in streaming settings where users can pose questions without anypre-annotated answers. Several studies (Zhang et al., 2022b; Li and Qiu, 2023b) have suggestedusing LLMs to generate answers for unlabeled data. They apply filtering techniques to determinethe quality of these generated answers, adding only those examples with high-quality answers to theretrieval corpus. The most widely used filtering technique is based on self-consistency (Wang et al.,2022c). This approach involves prompting the language model to generate multiple chains of thoughtand answers, then selecting the most common answer as the final response.

Cross-Domain In this setting, IID human-annotated demonstrations are not available for the testqueries, so one uses annotated demonstrations from other similar tasks (Cheng et al., 2023; Shi et al.,2022). Note that this is different from the mix-domain setting where part of the corpus is IID and partof it is not. For instance, Shi et al. (2022) describes a scenario where the goal is to parse a Chinesequery into SQL. However, the demonstrations are sourced from an English Text-to-SQL corpus, adomain with significantly more resources than the target domain. Shi et al. (2022) employs thishigh-resource data as the retrieval corpus. To adapt to the target domain during inference with a LLM,the target query is translated into the same language as the demonstrations. Nie et al. (2022) presentsa similar approach, retrieving demonstrations from high-resource domains to address low-resourcequeries. However, their retrieval pool consists of multiple high-resource sources

n the mix-domain setting (Wang et al., 2023a; Li et al.,2023b), the retrieval corpus is constructed from the combination of all tasks. At the inference time,given a question, the retriever will retrieve demonstrations from this mixed corpus; the demonstrationscan come from the same domain as the test question or from other tasks.

n this setting, an in-domain training set, independently and identically distribution(IID) with the test queries, is available and serves as the retrieval corpus. Most existing work takethe full training set as the corpus. However, to be more annotation efficient, Hongjin et al. (2022)uses only a subset M of the training set N which includes the most representative and diverse ones,where |M | << |N |. One question that remains unanswered from the work of Hongjin et al. (2022)is how the predictive performance is affected as a function of retrieving from a subset M insteadof the entire training set N . While there is no follow-up work to answer this question, the closestcomparison we find is the results in Ye et al. (2023a) where a similar setup as Hongjin et al. (2022) isused except that they use the entire training set as the retrieval corpus, and report lower performanceon the SST-5 dataset (compare the Figure 3 in Hongjin et al. (2022) and Table 3 in (Ye et al., 2023a)).While there might be other differences between the two setups that may affect the final performance,this comparison implies that retrieving from a carefully selected subset might have comparable resultsto retrieving from the entire training set.

This setting assumesthat training data related to a task is available, and thus can be used as the retrieval corpus

terative Retrieval The earlier retrieval strategies acquire each demonstration independently.However, in iterative retrieval, a retriever selects demonstrations based on both the query andpreviously retrieved demonstrations. This process starts with a single query, for which the retrieverfinds one best demonstration. The query is then augmented (e.g. combined with the demonstration) toretrieve the next demonstration. This step is iteratively executed k times to gather k demonstrations.The general idea is to select the demonstrations that can complement each other. An an example of awork from this categorym, Scarlatos and Lan (2023) train an LSTM retriever using a reinforcementlearning framework. During the inference phase, the retriever processes the input query to select thebest initial demonstration. It then generates a new query representation by integrating the query withprior demonstrations, specifically utilizing the hidden state representation from the LSTM model.This process of updating the query representation and obtaining subsequent demonstrations continuesiteratively until k demonstrations are retrieved

Clustering Retrieval To mitigate the issue of homogeneity in one-hot retrieval, clustering retrievalapproaches (Li et al., 2022; Zhang et al., 2022b; Li and Qiu, 2023b) categorize all demonstrationsinto k sub-groups aiming to group similar demonstrations together. Then given a query, the retrieverpicks the most similar demonstration from each sub-group resulting in a final set of k demonstrations.The core principle of clustering is to select a diverse range of demonstrations. Most of the work useSBERT Reimers and Gurevych (2019a) to encode the demonstrations (only the question or the entiredemonstrations) and then apply k-means for clustering.

One-hoc Retrieval This is the most basic retrieval strategy. To obtain k demonstrations, given aquery, the retriever ranks the demonstrations based on some scoring criteria and then selects the top-kdemonstrations. Thus, each demonstration is chosen independently of the others. This method isstraightforward and fast, however, it might not yield the best combination of k demonstrations asthese demonstrations might be homogeneous.

Levy et al. (2022) found thatdiversity and coverage are important when the model is unfamiliar with the output symbols space. Itis noteworthy that researchers have found that ICL benefits more from demonstrations with highercomplexity in some scenarios (Fu et al., 2022), where they define the complexity in terms of the querylength or reasoning steps

Beyond similarity, some work has found that the diversity of demonstrationsis important. The motivations for diversity include avoiding repetitive demonstrations (Zhanget al., 2022b), bringing different perspectives (Yu et al., 2023), and maximizing the demonstrations’coverage of the test query, in terms of covering either its words or syntactic structures (Levy et al.,2022).

Similarity involves selecting demonstrations most akin to the query and can be basedon language similarity (term matching or semantic matching), structural aspects (sentence structure,reasoning structure, etc.), or other criteria. Most studies focus on language similarity, with feweraddressing structural similarity, often due to the challenges in extracting a query’s structure in manytasks (Levy et al., 2022).

here are two primary retrieval objectives for selecting demonstrations: similarityand diversity.

emonstration Formatting: Various works have shown that the formatting and wording of theprompts can play a crucial role in the performance of the LLM (Jiang et al., 2020; Shin et al., 2020;Kojima et al.; Yang et al., 2023). For example, Kojima et al. show that simply adding Let’s thinkstep by step to the prompt makes LLMs reason step by step and solve substantially more problems,and Weller et al. (2023) show that adding According to Wikipedia to the prompt makes them morefactual. Moreover, Min et al. (2022b) shows that besides the text formatting, the label space and thedistribution of the input text in the demonstrations are also of immense importance

Traditionally, the same set of few-shot demonstrations is used on all queries, which can be suboptimalespecially when there are high variations among the queries. An alternative is to retrieve few-shotdemonstrations that are tailored to the current query. Previous work has shown that demonstrationretrieval leads to substantial improvements in the task metrics, compared to manually curated orrandomly selected demonstrations (Luo et al., 2023; Ye et al., 2023a). Furthermore, LLMs have beenshown to become less sensitive to the factors such as demonstration ordering (Section 2.2) whenretrieved demonstrations are used (Li et al., 2023b)

Chain of Thought (CoT): It has been shown that including a rationale for the answer significantlyimproves model performance, especially for models that are larger than a certain size (Suzgun et al.,2022). The rationale is commonly known as chain of thought (CoT) (Wei et al., 2022). In the case ofCoT prompting, the demonstrations are typically formatted as: