Decoder
A decoder generates an output sequence, typically one token at a time, using the encoded context and what it has generated so far.
Prerequisites
What a Decoder Does
A decoder generates output, one token at a time. At each step, it looks at everything generated so far — plus, in an encoder-decoder setup, the encoder's representation of the input — and predicts the single most likely next token. That new token is added to the sequence, and the process repeats.
Tokens Generated So Far
Running SequenceEverything the decoder has produced up to now.
Decoder
Reads SequenceConsiders what has been generated so far.
Predict Next Token
Single PredictionThe single most likely next token.
Append to Sequence
Sequence GrowsThe new token becomes part of the running output.
Repeat
One Token at a TimeThe whole process runs again for the next token.
Key Idea
This step-by-step process is exactly why streaming works — each token exists as soon as it's generated, well before the full response is complete.
Decoder-Only Models
Most modern chat-oriented LLMs are decoder-only: there's no separate encoder stage. The decoder reads the entire prompt as the beginning of the sequence and continues generating from there, using self-attention over everything that came before — the prompt and its own previously generated tokens alike.
Important
A decoder can only attend to earlier positions in the sequence, never later ones — this is what makes it suitable for generation, where future tokens genuinely don't exist yet.
Common Mistakes
Assuming a decoder generates the whole response at once
Generation happens one token at a time, with each new token conditioned on everything generated so far — this sequential process is fundamental to how LLMs work.
Assuming every model needs both an encoder and a decoder
Decoder-only architectures, without any separate encoder, power most modern chat-oriented LLMs.
Forgetting that a decoder can't see future tokens
At any point in generation, a decoder only has access to what came before it — this is a structural constraint, not a limitation of a particular model.
Interview Question
How does a decoder generate a response, and why does that make streaming possible?
A decoder generates output one token at a time — at each step, it looks at everything generated so far, along with the prompt, and predicts the single most likely next token, then repeats using the newly extended sequence. It can only attend to earlier positions, never later ones, which is a structural constraint that makes sense for generation, since future tokens genuinely don't exist yet. This step-by-step process is exactly why streaming works — each token is a real, complete output the moment it's generated, so an application can display it immediately instead of waiting for the entire response to finish.
What an interviewer may ask next
- Why can't a decoder attend to tokens that haven't been generated yet?
- How does the decoder's step-by-step generation process relate to streaming?
- What does it mean for a model to be "decoder-only"?
Explain It in 30 Seconds
A decoder generates output one token at a time, using everything generated so far — plus the prompt — to predict the next token, then repeating with the extended sequence. It can only attend to earlier positions, never future ones that don't exist yet. This step-by-step process is exactly why streaming works, since each token is a complete output the moment it's produced. Most modern chat-oriented LLMs are decoder-only, with no separate encoder stage at all.