less than 5%);

Table 9.

Raspberry Pi 4B, achieving 0.7 tokens/s for 7B models

prompt length of 4 tokens

Overall, using the same calibration and evalua-tion distribution works the best

Figure 5.

Table 4.

perplexity

For attention layers, wefuse QKV projections into a single kernel

We avoidwriting dequantized weights into DRAM by fusing dequan-tization kernels with the matrix multplication kerne

Middle: The generation stage is memory bound and has low arithmetic intensity. W4A16 quantization caneffectively improve the arithmetic intensity by 4×. Right: The amount of weight access is orders of magnitude larger than the amount ofactivation access. Thus, weight-only quantization is more effective for on-device LLMs.

we take a small calibration

he

Weempirically find that: (1) The expected error from Round(·)(denoted as RoundErr(·)) does not change

group/block

such a mixed-precision data type will make the system implementationdifficult.

activation magnitude

uantization maps a floating-point number into lower-bitintegers.

group-wiseINT4 quantization

e focus on thesecond setting in this work since it not only reduces thehardware barrier (requiring a smaller memory size) but alsospeeds up the token generation (remedies memory-boundworkload).

is

scaling up the salientchannels can reduce their relative quantization error

activation distribution

, let alone edge devices

educing the chance of databreaches

modalities

edge devices

For each application,

unordered future block indexes underthe same page as the current memory address

There are three significant advantagesof using the binary vector as input

ure for large language models, simultaneously achieving training parallelism,low-cost inference, and good performance. We theoretically derive the connectionbetween recurrence

workloads vary over time

The good deci-sion ratio allows us to run a simulation only onc

Figure 11

However, the impactin hugepage coverage loss to application working sets can be seenin an increase of 7% in TLB misses on 2-tier machines compared to1-tier machines, corroborating the causal explanation of the 1.5%impact.

hot age as number of scan periods during whichthe page was recently accessed

show-ing the observed range of STAR and the corresponding impact toapplication performance

#addresses

Figure 8: An overview of key components of our LSTM based prefetcher.

The model can be used to gen-erate predictions right away (e.g. after the first epoch oftraining), and while more epochs are completed, more ac-curate estimates can be generated

100K accesses as warmup,and then use the next 200K for training and the next 200

caps the vocabulary at the 2Kmost popular deltas

autocorrelation coef-ficients (ACFs

Last Level Cache (LLC)

predefined distance

confidence on the current and previ-ous patterns, which is the probability it assigns to the correctprediction

miss history length of 1

average similar examples, producing singlerepresentative cases

The LSTM takes >150 𝜇s per inferenceand >1 ms per example for training

interference

If more than two frequent kernels exist, weignore them entirely for prefetching.

Complex

single thread,

Intel’s Precise Event-BasedSampling (PEBS)

Delta

we first cluster the raw address space usingk-means