Transformer architecture

Decoder

softmax(QKᵀ / √dₖ)V

Same words, different roles

A developing inquiry. One idea at a time

We are thus led to examine how the parts of a sentence contribute to its interpretation. How do we determine which words belong together? What establishes the role of one word in relation to another? And how does the surrounding context help us resolve a relationship that would otherwise remain uncertain?

Definitions and proofs will establish properties of the constructions themselves; experiments and interactive examples will allow us to investigate their behavior and what they acquire through learning. Throughout, we shall return to the linguistic questions that motivated the development.

What draws me to this subject is the gap between how naturally we read and how difficult it is to say what a machine must represent. I want to make that difficulty visible, and then work through it carefully enough that the mathematics answers questions we have already learned to ask.

Language → structure → mathematics

Beginning with language, and letting the questions lead us toward the mathematics.

ame words. A different event.

An inquiry, one idea at a time ↓