further strengthen

Response Generation

Choosing the Reward Model

Coreset (CS)

Query by Committee (QBC)

Prediction Entropy (PE)

Margin Sampling (MS)

Front-loading (FR)

Instance Selection Criteria

retrain the modelfrom scratch

EcoAssistant contains three components.

Datasets,

interpolating it within a few samples

with smallanswer disagreemen

Strategy 2: LLM approximation.

Strategy 1: Prompt adaptation.

Pr[q(S(x)) ≥ q(L(x))]

Response Time Metric (RT )

Response Quality Metric (RQ)

nference Cost Metric (C)

The routing policy π : R → S maps each inferencerequest ri ∈ R to a node-LLM pair (nj , mk) ∈ S

the set of I inferencerequests

trained on synthetic data reflecting outputs from both the SLMand the cloud LLM.

cores each token

valuates its own outputs using a verifier prompt

active learning techniques

rained continuously on responsesfrom a more expensive teacher model (LLM)

determined by two methods: user feedbackand automated check

Starting with inexpensive models, it offloads thequery to more expensive models like GPT-4 only if earlierresponses fail a reliability threshold.

trained on preference data

OptLLM

treated as a multi-armed bandit (MAB) problem

trained a BERT-style encoder using BARTscoreto assess query quality

Routoo

uses anLLM as a Performance Predictor,

performance and co

ZOOTER

reward distillation

Tryage

using a Q-learning-inspired approach

ρk · Load(Mk)

minimize cost while maintainingaccuracy

ransformers relyexclusively on this specialized multi-head attention imple-mentation

transformers require minimal prior knowledge of prob-lem structure

This design enables efficient parallelization, mak-ing transformers particularly suitable for scaling to high-complexity models and processing large datasets.

routing and HI as optimization strategies formulti-LLM systems deployed under real-world constraints

compression, energy efficiency, and infrastructureoptimization

At the system level

Energy efficiency

Efficient inference for LLMs

1d

min(di−di,tj , dj )∑k=1(pj + k))

the decodinglatency for these running requests increases due to the addi-tional load from new requests.

(pi + di,t)

he prefill phase of the incomingrequests will block the running requests

k2,n

k1,n

We then map the arrived request node embeddingGj,(L−1)t as the input of the DRL agent.

thegraph structure and semantic relation between requests andedge experts will be lost.

wedefine the raw global state features f t as follows

constantly monitor runningrequests, waiting requests, and their GPU memory utilization.

both the workload on each edge expert andtheir GPU memory utilization

A key insight is that the running requests can continuouslyprovide GPU memory utilization information, and the waitingrequests indicate their increased latency and future contention.

we use a special token < extra token n >to represent edge experts and add this special token beforethe request input text

Additionally, during the LLMrouting decision process, we only need to roughly estimatea general range, such as an approximate generation scoreinterval or output length interval, rather than exact values.

it is impossible to obtain

The expert-related features, on the other hand, focuson the request features related to the edge expert handlingthe reques

The operational featurescapture the immediate output metrics of request

oS-awarereward based on this estimator

action impact estimator to assess the effects ofrouting decisions on overall QoS

average latency per token

the BERTScore metric

interference among reques

first directed to an Edge Access Point (eAP)

several critical issues