Optimization problem description. The first part is the text description of the optimization problem,including the objective function and solution constraints. For example, for prompt optimization,the LLM can be instructed to “generate a new instruction that achieves a higher accuracy”, and wedenote such instructions in the meta-prompt as meta-instructions. We can also provide customized

illustrates the overall framework of OPRO. In each optimization step, the LLM generatescandidate solutions to the optimization task based on the optimization problem description andpreviously evaluated solutions in the meta-prompt. Then the new solutions are evaluated and added tothe meta-prompt for the subsequent optimization process. The optimization process terminates whenthe LLM is unable to propose new solutions with better optimization scores, or a maximum numberof optimization steps has reached

eachoptimization step in our work generates new prompts that aim to increase the test accuracy based ona trajectory of previously generated prompts, instead of editing one input prompt according to naturallanguage feedback (Pryzant et al., 2023) or requiring the new prompt to follow the same semanticmeaning (Zhou et al., 2022b)

The meta-prompt contains two core pieces of information. The first piece ispreviously generated prompts with their corresponding training accuracies. The second piece is theoptimization problem description, which includes several exemplars randomly selected from thetraining set to exemplify the task of interest

wever, the large and discrete prompt space makes itchallenging for optimization, especially when only API access to the LLM is available. Followingprior work on continuous and discrete prompt optimization (Lester et al., 2021; Li & Liang, 2021;Zhou et al., 2022b; Pryzant et al., 2023), we assume a training set is available to compute the trainingaccuracy as the objective value for optimizatio

the optimal prompt formats can be model-specific and task-specific (Ma et al., 2023;Chen et al., 2023c). Therefore, prompt engineering is often important for LLMs to achieve goodperformance (Reynolds & McDonell, 2021).

Specifically, we focus on natural language tasks where both the taskinput and output are texts. LLMs are shown to be sensitive to the prompt format

Their ability to understand natural language lays out a new possibility for optimization: instead offormally defining the optimization problem and deriving the update step with a programmed solver,we describe the optimization problem in natural language, then instruct the LLM to iteratively generatenew solutions based on the problem description and the previously found solutions. Optimizationwith LLMs enables quick adaptation to different tasks by changing the problem description in theprompt, and the optimization process can be customized by adding instructions to specify the desiredproperties of the solutions.

We first showcaseOPRO on linear regression and traveling salesman problems, then move on to ourmain application in prompt optimization, where the goal is to find instructionsthat maximize the task accuracy

In this work, we propose Optimization by PROmpting(OPRO), a simple and effective approach to leverage large language models (LLMs)as optimizers, where the optimization task is described in natural language. Ineach optimization step, the LLM generates new solutions from the prompt thatcontains previously generated solutions with their values

In the decoding phase, we employ a greedy spansection (Zaratiana et al., 2022) that selects en-tity spans based on matching scores, to ensuretask/dataset specific constraints. This strategy is ap-plied independently to each sentence. Only, spans(i, j) with matching scores φ(i, j, c) > 0.5 are con-sidered for selection.Flat NER: The algorithm chooses the highest-scoring non-overlapping span and continues thisprocess until all spans are evaluated.Nested NER: Similar to Flat NER, but the algo-rithm allows selection of fully nested spans withinother entities while still avoiding partial overlaps.Algorithm Efficiency: The decoding is imple-mented using a priority queue for spans, ensuringan O(n log n) complexity, with n being the numberof candidate spans.

our objective is to optimize modelparameters to enhance the matching score for cor-rect span-type pairs (positive pairs) and reduce itfor incorrect pairs (negative pairs). A span (i, j)paired with an entity type t forms a positive pair(s ∈ P) if the span is labeled with type t in the train-ing data. Otherwise, it is a negative pair (s ∈ N ).The training loss for an individual example, com-prising spans S and entity types T , is defined as:LBCE = − ∑s∈S×TIs∈P log φ(s)+Is∈N log (1 − φ(s))(3)The variable s represents a pair of span/entitytype and I is an indicator function, which returns1 when the specified condition is true and 0 oth-erwise. This loss function corresponds to binarycross-entropy

To evaluatewhether a span (i, j) corresponds to entity type t,we calculate the following matching score:φ(i, j, t) = σ(STij qt) ∈ R (2)In this equation, σ denotes a sigmoid activationfunction. As we train with binary cross-entropyloss (see next sec. 2.2), φ(i, j, t) can be interpretedas the probability of the span (i, j) being of type t.

The computation of all span represen-tations can be easily parallelized. Moreover, we setan upper bound to the length (K=12) of the span inorder to keep linear complexity, without harmingrecall.

he entity representa-tion is computed by refining the initial represen-tation p using a two-layer feedforward network,resulting in q = {qi}M −10 ∈ RM ×D. The repre-sentation of a span starting at position i and endingat position j in the input text, Sij ∈ RD, is com-puted as:Sij = FFN(hi ⊗ hj ) (1)Here, FFN denotes a two-layer feedforward net-work, and ⊗ represents the concatenation operation.

h = {hi}N −10 ∈ RN ×D denotes the representationof each word in the input text. For words tokenizedinto multiple subwords, we use the representationof the first subword, which is a standard choice inthe NER literature

Let p = {pi}M −10 ∈ RM ×D represent the en-coder’s output for each entity type, correspondingto all the [ENT] token representation

put format The input to our model comprises aunified sequence combining entity types (expressedin natural language) and the input text from whichentities are to be extracted. The input format is asfollows:Human: What describes organization in the text ?Assistant: [‘Mcgill University’]Bidirectional LMs(BERT, DeBERTa)(0,1, person)(4,5, organization)[ENT] person [ENT] location [ENT] organization [SEP]+Alain Farley works at McGill Universitya) UniNER (prev) : Prompting LLM for Open NER.b) GLiNER (Ours): Prompting BiLM for Open NER.[ENT] [ENT] ... [ENT] [SEP] ...t0 t1 tM−1 x0 x2 xN−1[ENT] token represents a special token placedbefore each entity type and the [SEP] token func-tions as a delimiter, separating the sequence of en-tity types from the input text. They are initializedrandomly at the start of training

Our model has threemain components: i) a pre-trained textual encoder(a BiLM such as BERT), ii) a span representationmodule which computes span embeddings from to-ken embeddings, iii) an entity representation mod-

turally solves the scalabilityissues of autoregressive models and allows for bidi-rectional context processing, which enables richerrepresentations

n our work, we propose a model that addressesthe above-mentioned problems. Instead of relyingon large autoregressive models, we utilize smaller-scale Bidirectional Language Models (BiLM), suchas BERT (Devlin et al., 2019) or deBERTa (Heet al., 2021). The core concept of our model in-volves treating the task of Open NER as matchingentity type embeddings to textual span represen-tations in latent spac

While these works have achievedremarkable results, they present certain limitationswe seek to address: They use autoregressive lan-guage models, which can be slow due to token-by-token generation; Moreover, they employ largemodels with several billion parameters, limitingtheir deployment in compute-limited scenarios.Furthurmore, as NER is treated as a text gener-ation problem, the generation of entities is donein several decoding steps, and there is no way toperform the prediction of multiple entity types inparalle

powerfulLLMs typically consist of billions of parametersand thus require substantial computing resources.Although it is possible to access some LLMs viaAPIs (OpenAI, 2023), using them at scale can incurhigh cost

Traditional NER models arelimited to a predefined set of entity types. Expand-ing the number of entity types can be beneficial formany applications but may involve labeling addi-tional datasets

veraging a bidirectional transformer en-coder, our model, GLiNER, facilitates parallelentity extraction, an advantage over the slowsequential token generation of LLMs. Throughcomprehensive testing, GLiNER demonstratestrong performance, outperforming both Chat-GPT and fine-tuned LLMs in zero-shot evalua-tions on various NER benchmarks

Data construction prompt. Fig. 6 shows theprompt used for Chinese distillation data construc-tion. We follow Zhou et al. (2024) to design theprompt for Chinese data construction. We adoptthe data construction prompt of Pile-NER-type 3,since it shows the best performance as in (Zhouet al., 2024).Figure 6: Data construction prompt for Chinese opendomain NER.Data processing. Following (Zhou et al., 2024),we chunk the passages sampled from the Sky cor-pus4 to texts of a max length of 256 tokens andrandomly sample 50K passages. Due to limitedcomputation resources, we sample the first twentyfiles in Sky corpus for data construction, since thesize of the entire Sky corpus is beyond the pro-cessing capability of our machines. We conductthe same data processing procedures including out-put filtering and negative sampling as in UniNER.Specifically, the negative sampling strategy for en-tity types, is applied with a probability proportionalto the frequency of entity types in the entire con

ference with out-domain examples. Duringinference, since examples from the automaticallyconstructed data is not aligned with the domainsand schemas of the human-annotated benchmarks,we refer to them as out-domain examples. Fig. 4shows the results of inference with out-domain ex-amples using diverse retrieval strategies. We usethe model trained with NN strategy here. After ap-plying example filtering such as BM25 scoring, in-ference with out-domain examples shows improve-ments compared to the baseline, suggesting theneed of example filtering when implementing RAGwith out-domain examples

Training with diverse retrieval strategies. Fig.3 visualize the results of training with various re-trieval strategies. We conduct inference with andwithout examples for each strategy, and set the re-trieval strategy of inference the same as of training.The most straight forward method NN shows bestperformances, suggesting the benefits of semanti-cally similar examples. Random strategy, though in-Figure 4: Impacts of inferece with out-domain examplesusing various retrieval strategies. The average F1 valueof the evaluated benchmarks are reported. w/o exmp.means inference without example. Applying examplefiltering strategy such as BM25 filtering benefits RAGwith out-domain examples.Figure 5: Impacts of inference with in-domain examples.The average F1 value of the evaluated benchmarks arereported. N -exmp. means the example pool of size N .Sufficient in-domain examples are helpful for RAG.ferior to NN, also shows improvements, indicatingthat random examples might introduce some gen-eral information of NER taks to the model. Mean-while, inference with examples does not guaranteeimprovements and often hurt performances. Thismay due to the differences of the annotation schemabetween the automatically constructed data and thehuman-annotated benchmarks

Diverse retrieval strategies. The followingstrategies are explored in the subsequent analysis.(1) Nearest neighbor (NN), the strategy used in themain experiments, retrieves k nearest neighborsof the current sample. (2) Nearest neighbor withBM25 filter (NN, BM), where we apply BM25 scor-ing to filters out NN examples not passing a prede-fined threshold. Samples with no satisfied exam-ples are used with the vanilla instruction template.(3) Diverse nearest neighbor (DNN), retrieves Knearest neighbors with K >> k and randomly se-lects k examples from them. (4) Diverse nearestwith BM25 filter (DNN,BM), filters out DNN exam-ples not reaching the BM25 threshold. (5) Random,uniformly selects k random examples. (6) Mixednearest neighbors (MixedNN), mixes the using ofthe NN and random retrieval strategies with theratio of NN set to a.

We explore the impacts of diverse retrieval strate-gies. We conduct analysis on 5K data size for costsaving as the effect of RA-IT is consistent acrossvarious data sizes as shown in Section 3.4. Wereport the average results of the evaluated bench-marks here

The main results are summarized in Table 1 and2 respectively. We report the results of inferencewithout examples for RA-IT here, since we foundthis setting exhibits more consistent improvements.The impacts of inference with examples are studiedin Section 3.5. As shown in the tables, RA-ITshows consistent improvements on English andChinese across various data sizes. This presumablybecause the retrieved context enhance the model

We conduct a preliminary study on IT data effi-ciency in targeted distillation for open NER byexploring the impact of varous datas sizes: [0.5K,1K, 5K, 10K, 20K, 30K, 40K, 50K]. We use vanillaIT for preliminary study. Results are visualized inFig. 2. The following observations are consistentin English and Chinese: (1) a small data size al-ready surpass ChatGPT’s performances. (2) Perfor-mances are improving as the data sizes increased to10K or 20K, but begin to decline and then remainat a certain level as data sizes further increased to50K. Recent work for IT data selection, Xia et al.Figure 2: Preliminary study of IT data efficiency foropen NER in English (left) and Chinese (right) scenar-ios, where the training data are Pile-NER and Sky-NERrespectively. Average zero-shot results of evaluatedbenchmarks are illustrated. The performance does notnecessarily improve as the data increases.(2024); Ge et al. (2024); Du et al. (2023) also findthe superior performances of only limited data size.We leave selecting more beneficial IT data for IEas future work. Accordingly, we conduct mainexperiments on 5K, 10K and 50K data sizes

Training data: For English, we use thetraining data Pile-NER released by Zhou et al.(2024). For Chinese, we use the training data Sky-NER constructed in this paper as described in Sec-tion 3.2. We use LoRA (Hu et al., 2021) to trainmodels. Retrieval: We adopt GTE-large2 (Liet al., 2023) to generate text embeddings and setk = 2 in main experiments. Evaluation: Wemainly focus on the zero-shot evaluation. ForEnglish, we adopt benchmarks CrossNER, MIT-Movie and MIT-restaurant following Zhou et al.(2024). For Chinese, we collect eight benchmarksacross diverse domains, of which details are in Ap-pendix D. We report micro-F1 value

Retriever. We use sentence embedding-based re-trieval and adopt cosine similarity as our similaritymetric. We retrieve the k nearest neighbors as con-text. We also investigate various retrieval strategiesfor both training and inference stages

RA-IT. We explore an alternative way to conductIT in targeted distillation: we introduce RA-IT, acontext-enhanced tuning approach, of which theoverview is in Fig. 1. In our RA-IT approach,each data is augmented with a retrieved context,which consists of k semantically similar exam-ples retrieved from the training dataset. The re-trieved context is prepended to the original conver-sation, forming the retrieval augmented instruction.By fine tuning LMs in this recipe, we equip theLMs with the ability to generate NER answer withon-demand RAG. This means we could flexiblyadapting LMs to different scenarios by determin-ing whether to use RAG during inference based onthe specific characteristics of the scenario.

Vanilla IT. The original instruction tuning tem-plate used in targeted distillation is shown in thebottom part of Fig. 1, which we refer to as VanillaIT, where each passage and its associated entityoutput are converted into a multi-turn conversation.

reliminary: Targeted Distillation. We followUniNER (Zhou et al., 2024) to conduct our studyin the setting of targeted distillation, where theysuccessfully distill the strong capability of Chat-GPT in open NER into smaller models, without anyhuman-annotated data. The pipeline is as follows:(1) Data construction. They sample inputs froma large corpus across diverse domains, then useChatGPT to automatically generate NER outputs.(2) Distillation. After obtaining the automaticallyconstructed data, they apply IT to distill the openNER capability of ChatGPT into smaller models

1) Weempirically study the RA-IT framework for openNER. We prepare the retrieval augmented instruc-tion data with semantically similar examples. Weconduct thorough experimental analysis to studythe impact of various retrieval strategies. (2) We

(1) RA-ITachieves consistent improvements on various datasizes, suggesting the need for context-enhancedfine-tuning. (2) Retrieving semantically similar ex-amples benefits the most for training among variousretrieval strategies. Random retrieval also exhibitsimprovement but shows inferior performance tosimilar examples. (3) Retrieving out-domain ex-amples for inference requires applying examplefiltering strategies to achieve improvements. Pro-viding in-domain examples benefits inference.

our RA-IT approach, for each training sample,we retrieve semantically similar examples from thetraining dataset and prepend them to the original in-struction, forming the context-enhanced instruction.We also explore the impacts of diverse retrievalstrategies. Moreover, we construct a Chinese ITdataset for open NER and evaluate our methodin both English and Chinese scenarios. We con-duct thorough experiments across various data sizesand obtain the following key finding

The previous work UniNER (Zhou et al., 2024)distills the strong capability of ChatGPT in openNER into smaller models through IT without anyhuman-annotated data. We follow this line andinvestigate RA-IT under this targeted distillationsetting. Other works of IT for IE like Sainz et al.(2024); Li et al. (2024) using code-style instructiondata, are orthogonal to this work since RA-IT canbe integrated into various instruction styles.

Inthis paper, we explore Retrieval AugmentedInstruction Tuning (RA-IT) for IE, focusingon the task of open named entity recognition(NER). Specifically, for each training sample,we retrieve semantically similar examples fromthe training dataset as the context and prependthem to the input of the original instruction.

n summary, our contributions are three-fold: (i)We propose a framework CLUSTERLLM that uti-lizes sentence relations predicted from API-basedLLMs to guide clustering. Furthermore, it allowsusers to provide textual instructions and/or few-shot annotations to specify preferences on cluster-ing. (ii) In order to reduce API-queries, we proposea novel entropy-based sampling strategy to find themost informative triplets. Additionally, we utilizepairwise data sampled from hierarchical cluster-ing to determine cluster granularity. (iii) Extensiveexperiments show that our proposed method canimprove clustering performance at ∼$0.2 for per-spective and ∼$0.4 for granularity with GPT-3.5.

n Stage 2, we first obtain the cluster hierarchythat starts from instance-level clusters and itera-tively merge two closest clusters until the entiredataset. And then we prompt LLMs to determinecluster granularity with a few annotated data pairsas demonstrations. We construct the data pairsto prompt by sampling from two clusters that aremerged at each step of hierarchical clustering, sothat they cover a wide range of granularities. Andthe final decision is made by measuring consistencybetween each level of clustering and predictions.

In Stage 1, we prompt LLMs with a triplettask that predicts which one of the two candidatechoices is closer to anchor instance to understandthe user-preferred perspectives. We choose thistriplet task because (a) it is irrelevant with clustergranularity and (b) the produced triplets can fine-tune small embedder towards the right perspective.In order to improve sample efficiency, we furtherpropose entropy-based triplet sampling to find themost informative triplets. Specifically, we first cal-culate entropy for each instance based on clusterassignment probabilities, and then identify thosewith highest entropy. Two candidate choices arethen sampled from its nearest clusters to guaranteethey are close enough to the ancho

We propose CLUSTERLLM, a framework thatutilizes LLM to guide a small embedder for findingtext clusters with a low cost, as shown in Figure 1.It comprises two stages that are specially designedfor two aspects of clustering: (1) perspective, i.e.,the grouping criterion such as topic, intent and emo-tion and (2) granularity, i.e. the scope of clusters

n this paper, we provide insights on the ques-tion: Can we leverage API-based LLMs to guidetext clustering efficiently? We attack this challeng-ing question by drawing inspiration from an obser-vation that humans represent an instance throughcomparing with others

State-of-the-artlarge language models (LLMs) such as recent GPTseries (Brown et al., 2020; Ouyang et al., 2022;OpenAI, 2023) have demonstrated extraordinarylanguage capabilities for various NLP applicationshowever, these GPT models can only be utilizedthrough the APIs without accessible embeddingvectors for clustering. Hence, LLMs cannot bedirectly applied on text clustering tasks

ext clustering, as a fundamental task in natural lan-guage processing (NLP), has a wide spectrum ofapplications, such as identifying public perceptionfrom social media (Park et al., 2022), analysingcause of accidents (Xu et al., 2022), and detectingemerging research topics (Martínez et al., 2022). Acommon practice for text clustering is to apply clus-tering algorithms (MacQueen, 1967; Zhang et al.,∗ Corresponding author.1The cost is calculated with gpt-3.5-turbo.Texts CABTraditional Text ClusteringChatGPT(API-based)🔒ClusterLLMCABTexts CABChatGPT(API-based)🔒A should be closerto C than B🧐Not Applicable ⛔Embedder(Instructor,E5,GTR ...)Embedder(Instructor,E5,GTR ...)Figure 1: LLMs like ChatGPT are not applicable for textclustering directly because of the inaccessible embed-dings. CLUSTERLLM resolves the dilemma by leverag-ing LLM as a guide on text clustering.2021a) on top of pre-trained embedders (Muen-nighoff et al., 2022; Wang et al., 2022; Su et al.,2022) which could achieve higher performancewith better pre-training quality

Due to the diversity of possibilities in human lan-guage, it is rare for the same idea to be expressedidentically in multiple documents unless one ex-pression is derived from the other, or both are quot-ing from a shared source. This observation moti-vates deduplicating exact substrings. We call ourapproach EXACTSUBSTR. When two examplesxi and xj share a sufficiently long substring (thatis, a substring for which xa..a+ki = xb..b+kj ), thatsubstring is removed from one of them. Basedon statistical analyses (§B), we select k = 50 to-kens as the minimum matching substring length.3

We introduce two complementary methodsfor performing deduplication. First, using a suf-fix array (Manber and Myers, 1993), we removeduplicate substrings from the dataset if they oc-cur verbatim in more than one example. Second,we use MinHash (Broder, 1997), an efficient algo-rithm for estimating the n-gram similarity betweenall pairs of examples in a corpus, to remove entireexamples from the dataset if they have high n-gramoverlap with any other example

n our research, we do not focus on the impact ofduplicate text in pretrained models on downstreambenchmark tasks; instead we address how duplicatetext in the LM training and validation sets impactsmodel perplexity and the extent to which generatedtext included memorized content

GPT-3(Brown et al., 2020, §5) did the reverse and re-moved downstream evaluation examples from theirtraining data by conservatively filtering out anytrain set examples with a 13-gram overlap withany evaluation example. Up to 90% of tasks wereflagged as potentially contaminate

Trinh and Le (2018, Appendix B) removeddocuments from their CommonCrawl-based trainset that overlapped substantially with the common-sense reasoning used for evaluatio

ontamination of downstream tasks. Whenmodels are trained on datasets constructed by crawl-ing the Internet, it is possible the model will trainon the test set of downstream target tasks

We propose two scalable techniques to detectand remove duplicated training data. Exact sub-string matching identifies verbatim strings that arerepeated. This allows us to identify cases whereonly part of a training example is duplicated (§4.1).Approximate full document matching uses hash-based techniques (Broder, 1997) to identify pairsof documents with high n-gram overlap (§4.2).

We run all instruction tuning experiments fromthe Hungarian pretrained model using 6 gigabytesof instruction tuning text data (2 billion tokens)and the same training settings. Each experimentis repeated 3 times with different random datasetsamples. For more details on instruction tuningdatasets or training settings see appendix C.2, C.4and D.2

Given the same total amount of training data, we tested varying the percentage of English data (50%,25% and 0%) in the English/Hungarian bilingual data mixture. All training is run for 30k steps. Wealso compare this to training a pure Hungarian model using only Hungarian data [31], a Hungariantokenizer, from scratch for 100k steps. All the training details can be found in appendix D.1.

We categorize all evaluation tasks into 4 categories. Multiple Choice, for this category we appendeach candidate answer to the prompt and pick the highest probability answer. Open-ended QuestionAnswering, where we let the model generate an answer for each question, and report the averageF1 score between the model output and the ground truth. Summarization, where we let the modelgenerate a summary and report the average ROUGE-2 score between the model output and groundtruth. Translation, where we let the model generate translated text and report the BLEU scorebetween the model output and the ground truth. When we report the score for each category, it is theaveraged score of all the evaluation tasks that we classified into that category in appendix E.

Training is done in a two stage pipeline. The first stage is adaptive pretraining (PT) where a basepretrained English 13B GPT-2 model (B) is continuously trained on a mixture composed of the newlanguage and English. Then, the adapted checkpoint is instruction tuned (IT) on a collection ofprompt completion pairs from the new language and English. For more information see appendixB,C

Once the datasets are prepared for both languages, we shuffle them at sample level, so that everybatch contains text from both languages during training. Note that in our experiments, we do notmake any further transformations to either the model or the datasets, after the data is preparedon each side, so that our study is orthogonal and complementary to existing proposed methods[24, 27, 28, 9, 12] focusing on training paradigm studies.

To adapt an existing tokenizer to a new language, tokens from the low resource language can beadded to the existing tokenizer’s vocabulary to improve its fertility. Fertility is defined as the averagenumber of tokens per word [22], and details about how we calculated it can be found in appendixA.1. In our work, instead of extending the tokenizer’s vocabulary, we replace the least frequenttokens from it with tokens from the new language. This way, we keep the model capability the sameby controlling the vocabulary and embedding table size. In particular, we train a BPE tokenizeron the new language with vocabulary size k and check the number of overlapping tokens o withthe original tokenizer. Then we replace the least important k − o non-overlapping tokens from theoriginal tokenizer with the new ones. We also reinitialize the corresponding embeddings in themodel. For more details see appendix A.2

fertility

We adapt an English-centric model to Hungarian and Thai, and our evaluations show that adding newtokens and mixing training data from both languages can retain the model’s English capabilities inaddition to improving the models ability to learn the new language. Some contemporary worksexplore similar, but far less efficient methods of training LLMs on low resource languages. [30]builds an English-Arabic bilingual LLM, but they train it from scratch; while [29] builds one forEnglish-Portuguese, but it does not optimize the tokenizer or mix the training data

How to efficiently encode the new language? Byte Pair Encoding (BPE) [15] tokenizers are com-monly used in LLMs including GPT[16, 17], Llama [18, 19] and BLOOM [1, 2]. These tokenizersare able to encode text at the byte level so that they can generalize to characters that are outsideof their vocabulary; this means that any BPE tokenizer can be used for all languages. However,the BPE tokenizer has poor tokenization efficiency if it was not trained on a given language. Forexample, the original English-centric GPT2 tokenizer with a vocabulary size of 50k needs to use3.8 times more tokens to encode Thai compared to a smaller tokenizer with a vocabulary size of 5kthat is trained on Thai. This will inevitably cost us 3.8 times more compute in both training andinference. Furthermore, it has been shown that models with sub-optimal tokenizers can also haveworse evaluation results [20, 21]. In our work, we show how to improve tokenizer fertility[22] byreplacing the least frequent tokens in the base model with tokens from the new language.How to avoid catastrophic forgetting? Many works have shown that when continuing to train aLLM on data from a new domain, it undergoes catastrophic forgetting of the original domain it wastrained on [23], and similar issues appear when training on a new language [23, 9, 24, 2, 25, 10, 26].Different training paradigms including instruction-align[24], MAD-X [27], (IA)3 [28] are proposed

Multilingual large language models have become prevalent recently [1, 2, 3, 4, 5, 6], and haveshown strong cross lingual knowledge and capability transfer [7, 8, 9, 10, 11, 12, 13]. However,these multilingual models tend to perform poorly on low-resource languages. On top of this, trainingmodels for low-resource languages from scratch is also challenging due to a lack of training data andprohibitive computational requirements. These challenges, along with the prevalence open sourcedEnglish models creates an interesting opportunity to see how they can be adapted to new languagesquickly, without wasting resources by pretraining from scratch. While prior work [9, 10, 11, 14, 13]has studied this concept, there are two important questions that warrant further investigation

When agents have different targets in a task, especially when the targets are adversarial, the task can become much more complicated. An example of such a task is a zero-sum game, where the total reward is fixed, and any reward gained by one agent results in an equal loss for another agent. A specific example can be found in MPE that in scenarios like simple_push, agent ONE is trying to gain more reward by getting closer to its target location while agent TWO gains reward by pushing agent ONE away from the target location. Moreover, the competitive-like mode can also be not so pure competitive. It can incorporate some cooperative agents’ relationships. This type of work mode is referred to as mixed mode. A representative task of mixed mode is MAgent, where agents are divided into several groups. Agents in the same group need to attack the enemy group cooperatively.

Another mode is collaborative, where agents can access individual rewards. Under this mode, the agents tend to work together, but the target varies between different agents. Sometimes individual rewards may cause some potential interest conflict. Collaborative task mode has less restriction and richer reward information for wilder algorithms development: il is a good solution for collaborative tasks, as each agent has been allocated an individual reward for doing a standard RL. Centralized Critic is a more robust algorithm family for collaborative tasks as the improved critic help agent coordinate using global information. Value Decomposition-based methods are still applicable for collaborative tasks as we can integrate all the individual rewards received into one (only the agents act simultaneously). Cooperative mode can also be transformed to collaborative as we can copy the global reward to each agent and treat them as an individual reward

The Cooperative-like task mode is prevalent in scenarios where agents are rewarded only when the team achieves a shared goal. This mode is considered a strict form of cooperation, where each agent cannot access its individual reward. In Cooperative tasks, agents must have a robust credit assignment mechanism to decompose the global reward and update their policies accordingly.

The current state of research on multi-agent reinforcement learning (MARL) is facing challenges regarding the diversity of multi-agent tasks and the categorization of MARL algorithms. These characteristics make it difficult to conduct a fair comparison of different algorithms and raise a question for researchers: should algorithms be developed for a specific task (task first) or for general tasks (algorithm first). This difficulty stems from the nature of multi-agent tasks, as well as the various learning styles and knowledge-sharing strategies.

Despite the simple task setting, however, the game is still very challenging as one agent needs to coordinate with another agent to achieve the highest reward: the joint action with the highest reward is not a good option from the view of the first agent if it is not willing to cooperate with another agent. Two-step Game evaluates whether an agent has learned to cooperate by sacrificing its reward for a higher team reward.

The first option for evaluating a new idea in MARL involves using a matrix and grid world task.One such example is the Two-step Game. In this task, two agents act in turn to gain the highest team reward. The task is very straightforward: two agents in the task the observation is a short vector with a length four two actions (A&B) to choose from

In the context of Multi-Agent Reinforcement Learning (MARL), a dataset corresponds to a collection of scenarios that comprise a single multi-agent task. Multi-agent tasks are customizable on a variety of aspects, such as the number of agents, map size, reward function, and unit status. This section provides a brief overview of the categories of multi-agent tasks, ranging from the simplest matrix game to real-world applications.

On the other hand, value decomposition-based algorithms can only be applied to cooperative and collaborative scenarios. These algorithms use a value decomposition technique to decompose the value function into individual value functions, one for each agent. The agents then learn their own policies based on their individual value functions. Since value decomposition-based algorithms do not use a centralized critic, they cannot be applied to competitive scenarios where the agents’ objectives conflict.

Centralized critic-based algorithms are applicable to all types of multi-agent tasks, including cooperative, collaborative, competitive, and mixed. These algorithms use a centralized critic to approximate the state-action value function, which enables agents to learn a policy that considers the actions of other agents and the global state

CTDE strikes a balance between coordination learning and deployment cost, making it a popular framework in MARL. Since multi-agent tasks involve numerous agents, learning a policy that aligns with the group target requires incorporating extra information from other sources. Thus, centralized training is the preferred choice. However, after training, the delivery of centralized information is too costly during deployment, leading to delays in decision-making. Furthermore, centralized execution is insecure, as centralized information can be intercepted and manipulated during transmission.

The CTDE framework, which stands for Centralized Training & Decentralized Execution, is a widely used approach in multi-agent reinforcement learning (MARL). In this setting, agents are trained together in a centralized manner where they can access all available information, including the global state, other agents’ status, and rewards. However, during the execution stage, agents are forced to make decisions based on their local observations, without access to centralized information or communication.

In a Partially Observable Markov Decision Process (POMDP), the system states are unobservable and probabilistically mapped to observations. The agent’s access to the system state is limited, and taking the same action can result in different observations. The observation is, however, still dependent on the system state. Hence, the agent must learn or hold a belief about its observation and learn a policy that accounts for all possible states.

In Wang et al. (2023a), the authors proposed to train the demonstrationretriever model with combined objectives: (1) knowledge distillation from the trained reward modelwhich can capture the preferences of LLMs over the retrieved candidates (2) InfoNCE-based con-trastive loss to incorporate the in-batch negatives. More specifically, the resulting loss function is asfollows:Lcombined = αLcont + Ldistill

nfoNCE Loss Another widely adopted training procedure is contrastive learning using the In-foNCE loss (Rubin et al., 2022; Cheng et al., 2023; Luo et al., 2023). When positive and negative11

The Determinantal Point Process model (Alex Kulesz, 2012) defines a proba-bility distribution over all the combinations of candidate demonstrations, giving high probability tosubsets that contain relevant and diverse items (Levy et al., 2022). It models diversity by incorporatingcross-candidate similarity scores, and models similarity via a per-candidate relevance score,

Wang et al. (2023a) and Li et al. (2023b) instead proposed to iterate the retrievermodel multiple times. More specifically, the retriever trained in iteration i will be employed toretrieve a new set of candidates for the subsequent iteration i + 1. Such an iterative training approachallows progressively improving retriever quality by mining better positive and hard negative examplesat each iteration

Distillation by KL Divergence Ye et al. (2023a) claims that although the InfoNCE loss has beenfound effective in training demonstration retrievers and can learn which examples might be superiorto others, it has the same treatment for all negative examples and the predicted scores from LLMare not fully utilized.

s an alternative to train a demonstration retriever using positive and negativeexamples, Shi et al. (2022) proposed to train the retriever by directly distilling the LLM’s scoringfunction. More specifically, the retriever model is designed to produce ranking scores that matchthe usefulness of a demonstration to help with the LLM inference; this is done by minimizing theKL-divergence between the top K examples score distribution from scoring LLM and the rankingscore distribution produced by the retriever

In the list-wise ranking objective, retriever can benefit from the full ranking of the candidate set to makeaccurate predictions for the most relevant demonstrations. However, obtaining the full rankinglist and calculating the loss function on top of it might be very expensive and time-consuming.Additionally, the model is trained to discern the relative preferences between examples withoutexplicitly determining whether an example can serve as an absolute good demonstration

he list-wise ranking approach looks at a list of candidate documents fora given query and tries to capture the correct ordering for it. Li et al. (2023b) proposed to inject theranking signals into the retriever using an approach inspired by LambdaRank (Burges, 2010

approach to collecting training data for demonstration retriever is to directlymeasure the similarity between the labels of the candidate demonstrations and the label of the query,and use this similarity as a proxy of the importance of a demonstration (Hu et al., 2022; Poesia et al.,2021).

nce a score is obtained, a retriever can be trained that predicts these scores directly (Ye et al.,2023a). Alternatively, the candidate demonstrations can be ranked for each query based on theirscores

ased on LLMs Signals A popular approach to collecting training examples is to use the su-pervisory signals from LLMs. In this case, a typical paradigm is to first employ some filteringmechanisms (Cheng et al., 2023) or unsupervised retrievers (e.g. BM25 and SBERT) (Luo et al.,2023) as the initial retriever, this step can help limit the pool size for mining the right training data.Then a scoring LLM, which serves as a proxy for the inference LLM, is used to score each candidatedemonstration d. Here the score is defined as s(e) = p(a|d, q) which is the conditional probability ofoutput answer a given the input query q and demonstration d. Another approach is to train a smallerreward model that can provide more fine-grained supervision for dense retrievers. For example, Wanget al. (2023a) proposed to finetune a cross-encoder model serving as a teacher model for training theretriever

Researchers thus have started to explore learning-based methods to further push theboundaries. A typical objective when designing a good demonstration retriever is: if an LLM finds ademonstration useful when being used as an illustrative example, the retriever should be encouragedto rank the demonstration higher

Pretrained Dual Encoder In the context of demonstration retrieval where the goal is to identifyrelevant examples for a given query, the query is typically a question, while the examples maycontain additional information such as answers, chains of thoughts, supporting knowledge, or evenfollow different patterns. Therefore, transforming them into a uniform embedding space to calculaterelevance might not be the most effective approach. In this case, LLM retrieval architectures suchas Dual Encoder that are pretrained on retrieval or question-answering tasks can better grasp theintricate relationships between complex logical concepts and reasoning processes by employingdifferent semantic embeddings for queries and candidates (Li and Qiu, 2023b). In practice, traininga dual-encoder can be highly expensive as it typically requires a large training corpus. Fortunately,there are publicly available pretrained retrievers, although not specifically optimized for few-shotretrieval tasks, already demonstrating success in helping LLMs to learn from the selected examples.Luo et al. (2023) studied applying GTR (Ni et al., 2021) to select semantically similar examples asdemonstrations, and empirically proved that this approach brought in better performance gain thanrandom fewshots for both PaLM (Chowdhery et al., 2023) and FLAN (Chung et al., 2022) models.GTR is a T5-based dual encoder model that is pretrained on the CommunityQA (Abujabal et al.,2019) and finetuned on the MS Marco dataset (Nguyen et al., 2016). Moreover, Khattab et al. (2022)reported results for employing ColBERTv2 (Santhanam et al., 2021) as the retrieval module in theirDEMONSTRATE–SEARCH–PREDICT (DSP) framework for ICL. ColBERTv2 is a state-of-artretrieval model that adopts the late interaction architecture (Khattab and Zaharia, 2020) and is trainedon the MS Marco dataset. In the proposed framework, it is used to retrieve both (i) related knowledgeduring the search stage and (2) top k similar examples as demonstrations.

Shi et al. (2022)extends the use case to cross-lingual few-shot retrieval in the Text to-SQL semantic parsing task, andthey use mSBERT (Reimers and Gurevych, 2019b), mUSE (Yang et al., 2019) and mT5 (Xue et al.,2020) as the baseline models for comparison. Other widely used baseline models for demonstrationretrieval include E5base (Wang et al., 2022b), SimCSE (Gao et al., 2021b). Instead of relying on“word matches” as in BM25, these sentence embedding similarity approaches can better capturesemantic similarity (

Sentence Embedding Similarity In this approach, queries and documents are encoded to thesame dense embedding space using an off-the-shelf sentence embedding model, and then similarityscores (e.g. cosine similarity) are calculated to rank the most relevant documents for each query.A rich collection of sentence embedding methodologies exists in the literature.

Term-based Similarity BM25 (Robertson et al., 2009) is one of the most popular term-basedscoring methods due to its simplicity and effectiveness in producing relevant results. It takes intoaccount both term frequencies and document lengths. It has been empirically demonstrated in variousworks (Luo et al., 2023; Rubin et al., 2022; Agrawal et al., 2022; Ye et al., 2023a; Dalvi et al., 2022)that using BM25 to select similar examples as few-shots in ICL can help improve the performance ofmany LLM inference tasks. While BM25 has become a standard baseline model in the field, it is notwithout its limitations. Due to its sole reliance on term frequency and document length, this approachmay overlook crucial aspects such as semantic meaning and sentence structure, potentially leading toinaccuracies in certain instances. Another drawback is that BM25 lacks the capability for fine-tuningin downstream tasks, making it less competitive compared to neural models which can be fine-tunedand customized for specific downstream tasks.

Free Form Corpus Another approach to deal with the lack of human-annotated data for similartasks is create pseudo-demonstrations from unstructured text. Toward this goal, Lyu et al. (2022)utilized the Demix dataset (Gururangan et al., 2022), which is not tailored for any specific task. Togenerate pusedo-demonstrations, a retriever selects the top-k most relevant sentences from the dataset.Subsequently, arbitrary labels are attached to each sentence to form the examples. Li et al. (2022)propose a synthetic question answering generation method to create QA pairs using the syntheticgenerated passages by an LLM

Unlabelled Queries with Automatically Generated Answers The previous three corpora allpresuppose the availability of human-annotated data. However, this assumption may not hold inreal-life scenarios, particularly in streaming settings where users can pose questions without anypre-annotated answers. Several studies (Zhang et al., 2022b; Li and Qiu, 2023b) have suggestedusing LLMs to generate answers for unlabeled data. They apply filtering techniques to determinethe quality of these generated answers, adding only those examples with high-quality answers to theretrieval corpus. The most widely used filtering technique is based on self-consistency (Wang et al.,2022c). This approach involves prompting the language model to generate multiple chains of thoughtand answers, then selecting the most common answer as the final response.

Cross-Domain In this setting, IID human-annotated demonstrations are not available for the testqueries, so one uses annotated demonstrations from other similar tasks (Cheng et al., 2023; Shi et al.,2022). Note that this is different from the mix-domain setting where part of the corpus is IID and partof it is not. For instance, Shi et al. (2022) describes a scenario where the goal is to parse a Chinesequery into SQL. However, the demonstrations are sourced from an English Text-to-SQL corpus, adomain with significantly more resources than the target domain. Shi et al. (2022) employs thishigh-resource data as the retrieval corpus. To adapt to the target domain during inference with a LLM,the target query is translated into the same language as the demonstrations. Nie et al. (2022) presentsa similar approach, retrieving demonstrations from high-resource domains to address low-resourcequeries. However, their retrieval pool consists of multiple high-resource sources

n the mix-domain setting (Wang et al., 2023a; Li et al.,2023b), the retrieval corpus is constructed from the combination of all tasks. At the inference time,given a question, the retriever will retrieve demonstrations from this mixed corpus; the demonstrationscan come from the same domain as the test question or from other tasks.

n this setting, an in-domain training set, independently and identically distribution(IID) with the test queries, is available and serves as the retrieval corpus. Most existing work takethe full training set as the corpus. However, to be more annotation efficient, Hongjin et al. (2022)uses only a subset M of the training set N which includes the most representative and diverse ones,where |M | << |N |. One question that remains unanswered from the work of Hongjin et al. (2022)is how the predictive performance is affected as a function of retrieving from a subset M insteadof the entire training set N . While there is no follow-up work to answer this question, the closestcomparison we find is the results in Ye et al. (2023a) where a similar setup as Hongjin et al. (2022) isused except that they use the entire training set as the retrieval corpus, and report lower performanceon the SST-5 dataset (compare the Figure 3 in Hongjin et al. (2022) and Table 3 in (Ye et al., 2023a)).While there might be other differences between the two setups that may affect the final performance,this comparison implies that retrieving from a carefully selected subset might have comparable resultsto retrieving from the entire training set.

This setting assumesthat training data related to a task is available, and thus can be used as the retrieval corpus

terative Retrieval The earlier retrieval strategies acquire each demonstration independently.However, in iterative retrieval, a retriever selects demonstrations based on both the query andpreviously retrieved demonstrations. This process starts with a single query, for which the retrieverfinds one best demonstration. The query is then augmented (e.g. combined with the demonstration) toretrieve the next demonstration. This step is iteratively executed k times to gather k demonstrations.The general idea is to select the demonstrations that can complement each other. An an example of awork from this categorym, Scarlatos and Lan (2023) train an LSTM retriever using a reinforcementlearning framework. During the inference phase, the retriever processes the input query to select thebest initial demonstration. It then generates a new query representation by integrating the query withprior demonstrations, specifically utilizing the hidden state representation from the LSTM model.This process of updating the query representation and obtaining subsequent demonstrations continuesiteratively until k demonstrations are retrieved

Clustering Retrieval To mitigate the issue of homogeneity in one-hot retrieval, clustering retrievalapproaches (Li et al., 2022; Zhang et al., 2022b; Li and Qiu, 2023b) categorize all demonstrationsinto k sub-groups aiming to group similar demonstrations together. Then given a query, the retrieverpicks the most similar demonstration from each sub-group resulting in a final set of k demonstrations.The core principle of clustering is to select a diverse range of demonstrations. Most of the work useSBERT Reimers and Gurevych (2019a) to encode the demonstrations (only the question or the entiredemonstrations) and then apply k-means for clustering.

One-hoc Retrieval This is the most basic retrieval strategy. To obtain k demonstrations, given aquery, the retriever ranks the demonstrations based on some scoring criteria and then selects the top-kdemonstrations. Thus, each demonstration is chosen independently of the others. This method isstraightforward and fast, however, it might not yield the best combination of k demonstrations asthese demonstrations might be homogeneous.

Levy et al. (2022) found thatdiversity and coverage are important when the model is unfamiliar with the output symbols space. Itis noteworthy that researchers have found that ICL benefits more from demonstrations with highercomplexity in some scenarios (Fu et al., 2022), where they define the complexity in terms of the querylength or reasoning steps

Beyond similarity, some work has found that the diversity of demonstrationsis important. The motivations for diversity include avoiding repetitive demonstrations (Zhanget al., 2022b), bringing different perspectives (Yu et al., 2023), and maximizing the demonstrations’coverage of the test query, in terms of covering either its words or syntactic structures (Levy et al.,2022).

Similarity involves selecting demonstrations most akin to the query and can be basedon language similarity (term matching or semantic matching), structural aspects (sentence structure,reasoning structure, etc.), or other criteria. Most studies focus on language similarity, with feweraddressing structural similarity, often due to the challenges in extracting a query’s structure in manytasks (Levy et al., 2022).

here are two primary retrieval objectives for selecting demonstrations: similarityand diversity.

emonstration Formatting: Various works have shown that the formatting and wording of theprompts can play a crucial role in the performance of the LLM (Jiang et al., 2020; Shin et al., 2020;Kojima et al.; Yang et al., 2023). For example, Kojima et al. show that simply adding Let’s thinkstep by step to the prompt makes LLMs reason step by step and solve substantially more problems,and Weller et al. (2023) show that adding According to Wikipedia to the prompt makes them morefactual. Moreover, Min et al. (2022b) shows that besides the text formatting, the label space and thedistribution of the input text in the demonstrations are also of immense importance

Traditionally, the same set of few-shot demonstrations is used on all queries, which can be suboptimalespecially when there are high variations among the queries. An alternative is to retrieve few-shotdemonstrations that are tailored to the current query. Previous work has shown that demonstrationretrieval leads to substantial improvements in the task metrics, compared to manually curated orrandomly selected demonstrations (Luo et al., 2023; Ye et al., 2023a). Furthermore, LLMs have beenshown to become less sensitive to the factors such as demonstration ordering (Section 2.2) whenretrieved demonstrations are used (Li et al., 2023b)

Chain of Thought (CoT): It has been shown that including a rationale for the answer significantlyimproves model performance, especially for models that are larger than a certain size (Suzgun et al.,2022). The rationale is commonly known as chain of thought (CoT) (Wei et al., 2022). In the case ofCoT prompting, the demonstrations are typically formatted as:

Diversity of Demonstrations: Another important factor in the success of few-shot learning isthe diversity of the demonstrations. Naik et al. (2023) propose DiversePrompting where for thequestion of a demonstration, an LLM is used to generate different ways of solving the problem,and then those solutions are used in the prompt. Zhang et al. (2022b) propose to select a diverseset of questions as few-shot examples. Ma et al. (2023) propose a fairness metric for selectingdemonstrations which encourages selecting diverse few-shot demonstrations that produce a nearuniform predictive distribution for a semantic-free input

Order of Demonstrations: The order of demonstrations has been shown to substantially affect themodel performance. For example, Lu et al. (2022b) show that on some tasks, the model performancecan range from near-random to state-of-the-art depending on the order of the prompts, and Zhao et al.(2021) show that answers appearing toward the end of the prompt are more likely to be predicted bythe model

umber of Demonstrations: LLMs generally benefit from more demonstrations, but as the numberof demonstrations increases the rate of improvement typically decreases (Brown et al., 2020; Ye et al.,2023b; Min et al., 2022b). Generation tasks have been shown to benefit from an increased number ofdemonstrations more than classification tasks (Li et al., 2023b). Toward increasing the number ofdemonstrations, one barrier is the maximum context size of the LLM. While the size of the contexthas been increasing over time with newer LLMs, it may still be problematic for datasets with longinput texts or classification datasets with many classes

Several works try to provide theoretical justifications and insights into how LLMs learn from a fewin-context demonstrations (Xie et al., 2021; Garg et al., 2022; Von Oswald et al., 2023). However,the exact reasons behind this capability are still largely unclear making it difficult to select optimalfew-shot demonstrations

This not only improves the efficiency and scalability of the learningprocess but also has been shown to reduce biases inherent in manual exampleselection. I

However, the model’sability to perform ICL is sensitive to the choice of the few-shot demonstrations.Instead of using a fixed set of demonstrations, one recent development is to retrievedemonstrations tailored to each input query

1) The biggest challenges of multi-intent detec-tion (MID) in the deployment is the heavy coderefactoring on a running dialogue system whichalready does a good job in single-intent detection.It motivates us to design DialogUSR, which servesas a plug-in module and eases the difficulties ofincremental development.2) Prior work on MID has higher cost of dataannotation and struggles in the open-domain or do-main transfer scenarios. Only NLU experts canadequately annotate the intent/slot info for a MIDuser query, and the outputs of MID NLU modelsare naturally limited by the pre-defined intent/slotontology. In contrast, DialogUSR datasets can be1Code and data are provided in https://github.com/MrZhengXin/multi_intent_2022.!"#=>ChechowH#I:;Chec[SP]thereOPI9:TranRST12DEChec[SP]speedRV$H%PChec=>TH%P:;Chec[SP]fromH%P:;Z[Chec[SP]there^"_abSabSStep1. Initial Query Collection+,-./012345678Check the high-speed train from Xiamen toNanjing on Friday afternoonTask-orientedQuery Datasetsfghi:+,A/j12klh-./0345678Hi, I wanna check the high-speed train thatdeparts from Xiamen and arrives in Nanjing onFriday afternoonSamplingSentenceSimplificationStep2. Follow-up Query Creation'(* +,-./012345678Check the high-speed train from Xiamen to Nanjing on Friday afternoon'K* 9:;<=>How long does it take'Q* +A/BC6DEFGCheck out the special cuisine thereStep3. Query Aggregation'(* +,-./012345678Check the high-speed train from Xiamen to Nanjing on Friday afternoon'K* 9:;<=>How long does it take'Q* +A/BC6DEFGCheck out the special cuisine theremccnPcV%JS"* +,-./0123456789:;<=>?@+A/BC6DEFGCheck the high-speed train from Xiamen to Nanjing on Friday afternoon,how long does the journey take, then check out the special food there.'K* 9:;<=>How long does it take'Q* +A/BC6DEFGCheck out the specialcuisine there'KonPb* 123456789:;<=>How long does it take to travel fromXiamen to Nanjing in high-speed train'QonPb* +A/456DEFGCheck out the special cuisine in NanjingStep4. Query CompletionFigure 2: The overview for the data collection proce-dure of DialogUSR. Firstly we sample initial queriesfrom task-oriented NLU datasets (Sec. 2.1), then wehire crowdsource workers to write follow-up queries(Sec. 2.2). To aggregate the annotated queries, we pro-pose text filler templates (marked in red, Sec. 2.3) andpost-processing procedure. Finally we ask annotatorsto recover the missing information in the incompleteutterances (marked in blue, Sec. 2.4).easily annotated by non-experts, and the derivedmodels are domain-agnostic in the sense that thelearned query splitting, coreference/omission re-covery skills are generic for distinct domains3) Presumably MID is more difficult than sin-gle intent detection (SID) given the same inten-t/slot ontology. From the perspective of task(re)formulation, DialogUSR is the first to converta MID task to multiple SID tasks (the philosophyof ’divide and conquer’) with a relatively low er-ror propagation rate, providing an alternative andeffective way to handle the MID task.

he anno-tators are instructed to write up to 3 subsequentqueries on what they need or what they would liketo know about according to the designated initialquery

we ask human an-notators to put themselves in the same position ofa real end user and imagine they are eliciting mul-tiple intents in a single complex user query whileinteracting with conversational agents

sample an initial query from twoChinese user query understanding datasets fortask-oriented conversational agents, namely SMP-ECDT2(Zhang et al., 2017) and RiSAWOZ3 (Quanet al., 2020). Then we ask human annotators to sim-plify the initial queries that have excessive length(longer than 15 characters), or are too verbose orrepetitive in terms of semantics

we propose com-plex dialogue utterance splitting and reformulation(DialogUSR) task with corresponding benchmarkdataset that firstly splits the multi-intent query intoseveral single-intent sub-queries and then recoverthe coreferred and omitted information in the sub-queries,

Tointegrate the multi-intent detection model into aproduct dialogue system, the developers wouldmake extra efforts in continuous deployment, i.e.technical support for both single-intent and multi-intent detection models, and system modifications,i.e. changes in the APIs and implementations ofNLU and other related modules

o handle multi-intent user queries, a straight-forward solution is to train a dedicated natural lan-guage understanding (NLU) system for multi-intentdetection

n the context of the TOD system, two crucialcomponents for measuring the success of a dia-logue are belief state and system response

he main contributions of our work can be sum-marized as follows:1. We design a prompt construction methodbased on domain and slot information.2. We proposed an adaptive prompt generationframework for the comprehensive black-box LLM-based TOD system.3. Experimental results demonstrate the effec-tiveness of our approach in enhancing the capabili-ties of LLMs.

e de facto way of utilizing black-box largelanguage models (LLMs) to perform variousdownstream tasks is prompting. However,obtaining suitable prompts for specific tasksis still a challenging problem. While exist-ing LLM-based methods demonstrate promis-ing performance in the task-oriented dialogue(TOD) task, they often require manual adjust-ment in prompt selection or focus solely on dia-logue understanding or generation. To addressthese issues, we propose an adaptive promptgeneration framework to fully unleash the po-tential of LLMs for the comprehensive TODsystem. Firstly, we design a trainable slot gen-erator (TSG) that can generate domain and slotinformation in the belief state, which serves asprior knowledge for subsequent prompt genera-tion. Next, we propose an adaptive prompt gen-erator (APG) that utilizes the prior knowledgeto generate prompts for the LLM, deriving thebelief state and system response of the dialoguefor evaluation. Finally, we evaluate our frame-work on the MultiWOZ 2.0 dataset. Extensiveexperiments demonstrate th

With the Meta NLG tasks defined above, we formulate themeta-learning objective of Meta-NLG as below:θM eta = M etaLearn(T1, ..., TK )= arg maxθ EiEDTi ,D′TiLD′Ti(fθ′i) (4)θ′i = Adapt(DTi , θ) = θ − α∇θ LDTi (fθ ) (5)

Low-resource Adaptation. To simulate the process ofadapting to a low-resource NLG task, the sizes of both sub-sets DTi and D′Ti , especially DTi , are set small. Therefore,when the model is updated on DTi as a part of the later meta-learning steps, it only sees a small amount of samples in thattask. This setup embeds the goal of low-resource adaptation.

Task Generalization. To generalize to new NLG tasks,Meta NLG tasks follow the same modality as the target task.For example, if our target task is to adapt to DA-utterancepairs in a new domain, then DA-utterance pairs in each Tiare sampled from the same source domain. We also consideradapting to new DA types in later experiments. In this case,DA-utterance pairs in each Ti have the same DA type. Thissetting merges the goal of task generalization.

Therefore, the first step is to construct a set of auxiliaryMeta NLG tasks (T1, ..., TK ) to simulate the low-resourcefine-tuning process. We construct a Meta NLG task Ti by:Ti = (DTi , D′Ti ) (3)DTi and D′Ti of each Ti are two independent subsets of DA-utterance pairs from high-resource source data Ds. DTi andD′Ti correspond to meta-train (support) and meta-test (query)sets of a typical meta-learning or few-shot learning setup, andTi is often referred to as a training episode. This meta setupwith both DTi and D′Ti in one Meta NLG task allows ourMeta-NLG algorithm to directly learn from different MetaNLG tasks. The usage of them will be elaborated later. MetaNLG tasks are constructed with two additional principles:

the idea of our Meta-NLG algorithm isto repeatedly simulate auxiliary Meta NLG tasks from Ds tomimic the fine-tuning process in Eq.(2).

uppose fθ is the base NLG model parameterized by θ, andwe have an initial θs pre-trained with DA-utterance pairsDs = {(dj , Yj )}j∈s from a set s of high-resource sourcetasks. When we adapt fθ to some low-resource task t withDA-utterance pairs Dt = (dt, Yt), the fine-tuning processon Dt can be formulated as follows:θ∗ = Adapt(Dt, θ = θs) = arg maxθ LDt (fθ )= arg maxθ∑(dt ,Yt )∈DtlogP (Yt|dt; θ) (2)The parameter θs will be used for initialization, and themodel is further updated by new observations Dt. The sizeof Dt in low-resource NLG tasks is very small due to thehigh annotation cost, therefore, a good initialization parame-ter θs learned from high-resource source tasks is crucial forthe adaptation performance on new low-resource NLG tasks.Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)3152

hieved state-of-the-art perfor-mance by directly optimizing the gradient towards a goodparameter initialization for easy fine-tuning on low-resourcescenarios. It introduces no additional architectures nor pa-rameters.

The optimization algorithm itself canbe designed in a way that favors fast adaption

The idea is to use an additional meta-learnerto learn to update the original learner with a few trainingexamples. [Andrychowicz et al., 2016] developed a meta-learner based on LSTMs. Hypernetwork [Ha et al., 2016],MetaNet [Munkhdalai and Yu, 2017], and TCML [Mishraet al., 2017] also learn a separate set of representations forfast model adaptation. [Ravi and Larochelle, 2017] proposedan LSTM-based meta-learner to learn the optimization algo-rithm (gradients) used to train the original network

The idea is to learn a metric space and thenuse it to compare low-resource testing samples to rich train-ing samples

A fundamental problem is “fastadaptation to new and limited observation data”

We formulate the problem froma meta-learning perspective, and propose a gen-eralized optimization-based approach (Meta-NLG)based on the well-recognized model-agnostic meta-learning (MAML) algorithm.

Inthis paper, we study NLG in a low-resource settingto generate sentences in new scenarios with handfultraining examples

As Meta-NLG is model-agnostic as long asthe model can be optimized by gradient descent, we could ap-ply it to any existing NLG models to optimize them in a waythat adapt better and faster to new low-resource tasks

Then, we proposed a generalizedNLG algorithm called Meta-NLG based on MAML by view-ing languages in different domains or dialog act types as sep-arate Meta NLG tasks.

nstead of casting the problem as model-based approaches, we propose a generalized optimization-based meta-learning approach to directly enhance the opti-mization procedure for the low-resource NLG task

Although promis-ing results were reported, we found that datasets used bythese methods are simple which tend to enumerate many slotsand values in an utterance without much linguistic variations.

Meta-NLG defines aset of meta tasks, and directly incorporates the ob-jective of adapting to new low-resource NLG tasksinto the meta-learning optimization process.

De-spite the recent success of neural approaches forNLG, they are typically developed for particulardomains with rich annotated training examples

Git - Học nghiêm túc một lần

fine-tune SC-GPT on limited amounts of domain-specific labels for adaptation

Finally, thesequentialized dialog act A′ is concatenated withits augmented response x′, and then fed into GPT-2.

We firstly pre-process dialog act A into a se-quence of control codes using the following format:A′ = [ I ( s1 = v1 , · · · sP = vP ) ] (4)Meanwhile, the output sequence x′ is pre-processed via appending x with a special start to-ken [BOS] and an end token [EOS].

To enablethe guidance of dialog act in response generation,we propose to continuously pre-train the GPT-2model on large amounts of annotated (dialog act,response) pairs. The pre-training dataset3 includesannotated training pairs from Schema-Guided Dia-log corpus, MultiWOZ corpus, Frame corpus, andFacebook Multilingual Dialog Corpus. The totalsize of the pre-training corpus is around 400k ex-amples

Existing methods for NLG can be broadly sum-marized into two major categories. (i) Template-1Semantically-Conditioned Generative Pre-Trainingbased methods require domain experts to handcrafttemplates for each domain, and the system fills inslot-values afterward (Cheyer and Guzzoni, 2014;Langkilde and Knight, 1998). Thus, the producedresponses are often adequate to contain the requiredsemantic information, but not always fluent and na-ture, hurting users’ experiences. (ii) Statistical lan-guage models such as neural networks (Gao et al.,2019) learn to generate fluent responses via train-ing from labelled corpus. One canonical model issemantically conditioned LSTM (SC-LSTM) (Wenet al., 2015b), which encodes dialog acts with one-hot representations and uses it as an extra feature toinform the sentence generation process.

ayer Transformer neural language model, trainedin three steps: (i) Pre-trained on plain text, similarto GPT-2 (Radford et al.); (ii) Continuously pre-trained on large amounts of dialog-act labeled utter-ances corpora to acquire the ability of controllablegeneration; (iii) Fine-tuned for a target domain us-ing very limited amounts of domain labels.

o simulatesuch a few-shot learning setting, we have devel-oped a new benchmark dataset, called FEWSHOT-WOZ, based on the MultiWOZ (Budzianowskiet al., 2018) and Cambridge NLG datasets (Wenet al., 2016a). F

In a typical task-oriented dialogsystem, the Natural Language Generation (NLG)module plays a crucial role: it converts a systemaction (e.g., often specified in a semantic form se-lected by a dialog policy) into a final response innatural language. Hence, the response should beadequate to represent semantic dialog actions, andfluent to engage users’ attention

Regression Objective Function. The cosine-similarity between the two sentence embeddingsu and v is computed (Figure 2). We use mean-squared-error loss as the objective function

Classification Objective Function. We con-catenate the sentence embeddings u and v withthe element-wise difference |u − v| and multiply itwith the trainable weight Wt ∈ R3n×k:o = softmax(Wt(u, v, |u − v|))where n is the dimension of the sentence em-beddings and k the number of labels. We optimizecross-entropy loss.

SBERT adds a pooling operation to the outputof BERT / RoBERTa to derive a fixed sized sen-tence embedding. We experiment with three pool-ing strategies: Using the output of the CLS-token,computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of theoutput vectors (MAX-strategy). The default config-uration is MEAN.

The field of text generation systems shifted from traditional approaches to statistical approacheswhere the focus was on exploiting patterns in text data and building models to make a predictionbased on the text it has see

Notably, we iden-tify three important areas of further research towards building more effective dialogue systems:1) incorporating larger context, including conversation context and world knowledge; 2) addingpersonae or personality in the NLG system; and 3) overcoming dull and generic responses thataffect the quality of system-produced responses. We provide pointers on how to tackle these openproblems through the use of cognitive architectures that mimic human language understanding andgeneration capabilities

Some of the early success in the field of language generation was building systems like Eliza(Weizenbaum, 1966) and PARRY

Featurization Firstly, the policy featurizes theuser input, system actions and slots.

Similar to the REDP, we do not use aclassifier to select a system action. Instead, we jointlytrain embeddings for the dialogue state and each of thesystem actions by maximizing a similarity function be-tween them

Vlasov etal. [2] introduced the Recurrent Embedding Dialogue Pol-icy (REDP) architecture. The ablation study in this workhighlighted that the improved performance of REDP isdue to an attention mechanism over the dialogue historyand a copy mechanism to recover from unexpected userinput. This modification to the standard RNN structureenables the dialogue policy to ‘skip’ specific turns in thedialogue history and produce an encoder state which isidentical before and after the unexpected input.

Topic disentanglement in task-oriented dialogueRecent work has attempted to produce neural architec-tures for dialogue policies which can handle interleaveddiscourse segments in a single conversation

n the example above, theuser might follow up with a further question like so thatused up my credit, right?. If the topic of refund creditshas been popped from the stack, this can no longer helpclarify what the user wants to know

The authors of RavenClaw argue forexplicitly tracking topics to enable the contextual inter-pretation of the user intents

While a stack naturally allows for sub-dialogues to behandled and concluded, the strict structure of a stackis also limiting

The assistant’s question Shall Iplace the order? prompts the return to the task at hand:completing a purchase. One model is to think of thesesub-dialogues as existing on a stack, where new topicsare pushed on to the stack when they are introduced andpopped off the stack once concluded

discourse segments, where a discourse seg-ment (or topic) is a set of utterances that directly re-spond to each other.

The proposed TED architecture should bethought of as a candidate building block for use in de-veloping state-of-the-art architectures in various dialoguetasks.

Interpretingsimple instructions like please turn on the lights is rela-tively straightforward, but to handle more complex tasks,these systems must be able to engage in multi-turn con-versations.

ith higherrequirements on product experience, actual di-alog scenarios become more complex, and DMneeds to be further improved. Traditional DMis usually built in a clear dialog script sys-tem (searching for matching answers, query-ing the user intent, and then ending the dia-log) with pre-defined system action space, userintent space, and dialog body. However, dueto unpredictable user behaviors, traditional di-alog systems are less responsive and have agreater difficulty dealing with undefined sit-uations. In addition, many actual scenariosrequire cold start without sufficient tagged di-alog data, resulting in high data cleansing andtagging costs. DM based on deep reinforce-ment learning requires a large amount of datafor model training. According to the experi-ments in many academic papers, hundreds ofcomplete sessions are required to train a dialogmodel, which hinders the rapid developmentand iteration of dialog systems.To solve the limitations of traditional DM,researchers in academic and industry circleshave begun to focus on how to strengthen theusability of DM. Specifically, they are workingto address the following shortcomings in DM:• Poor scalability• Insufficient tagged data• Low training efficiency

sign and removing the isolation between mod-ules. However, the end-to-end model placeshigh requirements on the quantity and qualityof data and does not provide clear modelingfor processes such as slot filling and API call-ing. This model is still being explored and is asyet rarely applied in the industry.

This modular system structure is highly in-terpretable, easy to implement, and applied inmost practical task-oriented dialog systems inthe industry. However, this structure is notflexible enough. The modules are independentof each other and difficult to optimize together.This makes it difficult to adapt to changing ap-plication scenarios. Additionally, due to theaccumulation of errors between modules, theupgrade of a single module may require theadjustment of the whole system

ask-oriented dialog systems are divided byarchitecture into two categories. One type isa pipeline system that has a modular struc-ture(Wen et al., 2016), as shown in Figure 1.It consists of four key modules:

Common dialog systems are divided into thefollowing three types: chatting systems, task-oriented dialog systems, and QA systems. In achatting systems, the system generates inter-esting and informative natural responses to al-low human-machine dialog to proceed(Serbanet al., 2017)

The di-alog state is obtained by directly calculat-ing the maximum conditional probability in-stead of the Bayesian a posteriori probabil-ity.

n recent years, with breakthroughs in deeplearning in the image, voice, and text fields,third-generation dialog systems built arounddeep learning have emerged.

The first-generation dialog systems weremainly rule-based. For example, the ELIZAsystem(Weizenbaum, 1966) developed by MITin 1966 was a psychological medical chatbotthat matched methods using templates.

At that time, re-inforcement learning was widely studied andapplied in dialog systems. A representative ex-ample is the statistical dialog system based onthe Partially Observable Markov Decision Pro-cess (POMDP) proposed by Professor SteveYoung of Cambridge University in 2005(Younget al., 2013)

Turing test. Topass this test, the machine had to commu-nicate with a real person so that this per-son believed they were talking to another per-son

Inthis paper, we survey recent advances andchallenges within three critical topics forDM: (1) improving model scalability to fa-cilitate dialog system modeling in new sce-narios, (2) dealing with the data scarcityproblem for dialog policy learning, and (3)enhancing the training efficiency to achievebetter task-completion performance

Given the dialog history, DM pre-dicts the dialog state and decides thenext action that the dialog agent shouldtake.

n most cases, there are two kinds of errors in the Vietnamese language:mistyped errors and misspelled errors [11]. Mistyped errors are errors that occurduring the typing process. The majority of these mistakes are caused by the