KV Cache

KV Cache in Transformer Models: A Comprehensive Summary

  • In the context of serving transformer models, the KV (Key-Value) cache is a mechanism used to store and reuse intermediate computations during the generation of a sequence of tokens, particularly in autoregressive models like GPT. This technique is one of the most commonly used tricks for speeding up inference with transformer-based models, especially large language models (LLMs).

KV Cache: Key-Value Cache

  • Key (K) and Value (V) Tensors: In a transformer model, each attention layer computes attention scores based on key (K) and value (V) tensors, which are derived from the input tokens. These tensors are used to calculate how much focus each token should have on the other tokens in the sequence.
  • Caching Self-Attention Values: During self-attention, the sequence of tokens is projected using three separate, linear projections: key projection, value projection, and query projection. The KV cache stores the results of the key and value projections for future decoding iterations to avoid recomputing them every time.

Autoregressive Decoding Process

  • Step-by-Step Process:
    1. Initial Sequence: Start with a sequence of textual tokens.
    2. Predict Next Token: Predict the next token.
    3. Update Input: Add this token to the input.
    4. Repeat: Repeat until the generation is finished.

Importance of KV Cache

  1. Efficiency:
    • Reduced Computation: By caching the key and value tensors, the model can reuse them in subsequent steps without recalculating them. This significantly reduces the computational overhead, especially for long sequences.
    • Faster Inference: Since the computation for previously generated tokens is bypassed, the overall inference time is reduced, allowing for faster token generation and real-time applications.
  2. Scalability:
    • Handling Long Sequences: For long sequences, recomputing the K and V tensors at each step would be prohibitively expensive. The KV cache allows the model to handle longer sequences more efficiently by storing and reusing past computations.
    • Memory Management: Efficiently managing the KV cache helps in maintaining a balance between memory usage and computational speed, crucial for deploying large transformer models in production environments.
  3. Practical Deployment:
    • Real-Time Applications: In applications like chatbots, real-time text generation, and interactive systems, the latency introduced by recomputing attention scores can be detrimental. The KV cache ensures that responses are generated quickly.
    • Resource Optimization: Efficient use of the KV cache can lead to better resource utilization on the hardware, such as GPUs or TPUs, which is essential for serving large-scale transformer models.

Why Not Cache the Query?

  • Query Matrix: The entries in the query matrix are only needed to compute the representations of prior tokens in the sequence, whose key and value representations are already stored in the KV cache. At each time-step, the new query input consists of the token at that time-step and all prior tokens, making it unnecessary to cache the query projections.

Updates to the KV Cache

  • During Decoding: Throughout autoregressive decoding, the key and value projections are cached. Each time a new token is added to the input, the new rows are computed as part of self-attention and added to the KV cache. The query projection for the new token is then used with the updated key and value projections to perform the rest of the forward pass.

Latency Optimization

  • Reduction in Latency: KV caching decreases the latency for generating the next token in an autoregressive setting starting from the second token. The prompt tokens are not cached initially, so the time to the first token is higher. However, as KV caching kicks in for subsequent tokens, the latency reduces, optimizing the overall response time.

Scaling to Multi-Head Self-Attention

  • Multi-Head Attention: While the explanation primarily considers single-head self-attention for simplicity, the same process applies to the multi-head self-attention used by LLMs. This involves performing the process in parallel across multiple attention heads.


  • In summary, the KV cache in transformer models is a critical optimization that enhances the efficiency and speed of sequence generation, making it a key component for deploying these models in real-world applications. The use of KV caching in autoregressive decoding processes, along with its role in latency optimization and scalability, makes it indispensable for serving transformer-based models efficiently.


If you found our work useful, please cite it as:

  title   = {Model Acceleration},
  author  = {Chadha, Aman},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}