MLE 基础学习 Transformer-开发者社区

能解释算法原理、给出必要的数学推导、估算计算与显存成本、描述 GPU 上的执行方式，并讨论真实系统中的 Trade-off。Amazon Applied Scientist 官方面试说明本身就强调 theoretical analysis、coding、knowledge depth/breadth 和 scalable implementation；而 NVIDIA 的 Transformer/Inference 文档则直接把 attention 的 memory traffic、Tensor Core、FlashAttention、prefill/decode 等作为优化核心

需要从四个层次建立知识体系：

Mathematical Foundations：为什么设计成这样？能否严格推导？有什么假设与局限？
Model Architecture：数据怎样流过 Transformer？不同架构的计算量、参数量和表达能力如何？
Hardware and Systems：算子如何映射到 GPU？瓶颈在 FLOPs、HBM 带宽、Kernel Launch 还是跨 GPU 通信？
Production Optimization：怎样通过量化、KV Cache、Batching、并行策略与服务调度，在准确率、延迟、吞吐、成本之间取得平衡？

模块 1：数学基础与高维几何（Q001–Q020）

| Q01 | Attention 为什么用点积并缩放 | 统计推导、数值稳定、GPU 计算 |

Attention works by comparing the Query (Q) of each word with the Key (K) of every word to get scores, turning those scores into weights using softmax, and then using the weights to mix the Values (V)
编辑编辑

编辑

| Q02 | Transformer 参数、FLOPs 与 Prefill 成本 | 容量估算、建模 |

编辑编辑

为啥只存KV

编辑

| Q03 | RoPE 与 128K 长上下文 | 相对位置、外推、质量 |

| Q04 | MHA / GQA / MQA 与 KV Cache | 内存容量、延迟 |

MLA is somewhat analogous to an autoencoder bottleneck because it compresses the hidden state into a low-dimensional latent representation and later projects that latent into key/value representations. However, it is not a VAE and it does not try to reconstruct the original hidden state. It is better understood as a learned low-rank factorization of the KV projections. Although it introduces additional projection matrices, the latent dimension is much smaller than the full KV dimension, so the parameter overhead can be modest or even lower. More importantly, model weights are stored once, while KV cache grows per token and per request, so trading a small amount of extra compute or parameters for a much smaller KV cache is highly beneficial for long-context, high-concurrency inference.

| Q05 | FlashAttention / Online Softmax | IO-aware 算法 |

| Q06 | Roofline、Arithmetic Intensity | Compute/Memory Bound |
| Q07 | Prefill vs Decode | 两阶段性能建模 |
| Q08 | Continuous Batching 与 Token Scheduling | SLO-aware 排队调度 |
| Q09 | Paged KV Cache 与碎片化 | GPU 内存管理 |
| Q10 | Prefix Cache 与多租户安全 | 缓存设计 |
| Q11 | Speculative Decoding | 推测验证、收益模型 |
| Q12 | INT4 / INT8 / FP8 / FP4 | 量化与硬件协同 |
| Q13 | KV Cache Quantization | 累积数值误差、容量 |
| Q14 | TP / PP / DP / EP | 多 GPU 通信与拓扑 |
| Q15 | MoE 路由、负载均衡 | 专家并行 |
| Q16 | TTFT / ITL / p99 与容量规划 | 性能 SLO |
| Q17 | Serving 性能突然下降如何排障 | 可观测性与实验 |
| Q18 | CUDA/Triton Kernel 优化 | Tiling、Fusion、Spilling |
| Q19 | Prefill–Decode Disaggregation | 集群架构取舍 |
| Q20 | 设计 70B 长上下文生产推理平台 | 综合系统设计 |

Q	Interview-ready answer
Q02. Transformer Parameters, FLOPs, and Prefill Cost	A Transformer’s parameter count is dominated by the projection layers and FFN. For a standard dense block, the Q, K, V, and output projections contribute roughly \(4d^2\), while the FFN contributes roughly \(2d d_{ff}\), typically much larger when \(d_{ff}\approx 4d\). During prefill, the main compute scales roughly as \(O(Sd^2 + S^2d)\): the linear projections and FFN scale linearly with sequence length, while attention scales quadratically in \(S\). For short or moderate contexts, GEMMs and FFN can dominate; for very long contexts, the \(S^2\) attention term becomes increasingly important. In production capacity planning, I would model both FLOPs and memory traffic rather than relying on parameter count alone.
Q03. RoPE and 128K Long Context	RoPE injects position information by rotating query and key vectors with position-dependent angles, so the inner product naturally encodes relative position differences. One advantage is that it works directly inside attention instead of adding a separate positional embedding. The challenge at very long context, such as 128K, is extrapolation: the model may encounter relative distances and frequencies far outside the training regime, which can degrade attention quality. Practical long-context models often use RoPE scaling, interpolation, frequency rescaling, or modified rotary schemes. The key interview point is that extending the context window is not just a memory problem; it is also a positional-generalization problem.
Q04. MHA / GQA / MQA and KV Cache	In MHA, every query head has its own key and value heads, which gives maximum flexibility but creates a large KV cache. GQA reduces the number of KV heads by allowing multiple query heads to share the same K and V heads, while MQA shares a single K/V head across all query heads. This directly reduces KV-cache memory and memory bandwidth during decode. Since autoregressive decode is often memory-bandwidth-bound, GQA and MQA can substantially improve throughput and latency. GQA is a common compromise because it preserves more modeling capacity than MQA while achieving much of the KV-memory saving.
Q05. FlashAttention / Online Softmax	FlashAttention computes the same exact attention mathematically, but reorganizes the computation to minimize HBM traffic. Instead of materializing the full \(S\times S\) attention matrix in global memory, it processes Q, K, and V in tiles that fit in on-chip SRAM or shared memory. Online softmax allows the softmax normalization to be computed incrementally across tiles while maintaining numerical correctness. The main gain is not reduced asymptotic FLOPs; the algorithm is still fundamentally \(O(S^2d)\). The gain comes from reducing memory reads and writes and fusing operations, which is why FlashAttention is called an IO-aware algorithm.
Q06. Roofline and Arithmetic Intensity	The roofline model tells us whether a kernel is compute-bound or memory-bandwidth-bound. Arithmetic intensity is the number of FLOPs performed per byte transferred from memory. If arithmetic intensity is low, performance is limited by memory bandwidth; if it is high enough, performance is limited by peak compute throughput. Large GEMMs usually have high arithmetic intensity and can be compute-bound, while decode attention and KV-cache operations often have low arithmetic intensity and are memory-bound. This framework is useful because it tells me whether to optimize FLOPs, data movement, fusion, tiling, or memory layout.
Q07. Prefill vs Decode	Prefill processes the entire prompt in parallel, so it involves large matrix multiplications and full attention across many input tokens. It tends to have high GPU utilization and is often more compute-intensive. Decode generates one token at a time and reuses the KV cache, so each step has much smaller GEMMs but must repeatedly read a growing amount of KV-cache data. As a result, decode is frequently memory-bandwidth-bound. This is why TTFT is mainly influenced by prefill performance, while inter-token latency is mainly influenced by decode performance.
Q08. Continuous Batching and Token Scheduling	Continuous batching dynamically admits and removes requests at token boundaries instead of waiting for an entire static batch to finish. This improves GPU utilization because requests with different sequence lengths can share the same decode iteration. The scheduler usually has to balance throughput, fairness, TTFT, inter-token latency, and p99 SLOs. A throughput-only scheduler may starve short requests behind long ones, so production systems often use priority, aging, token budgets, or prefill/decode-aware scheduling. The core idea is that LLM serving is a dynamic queueing problem, not just a static batching problem.
Q09. Paged KV Cache and Fragmentation	A paged KV cache divides KV memory into fixed-size blocks or pages instead of allocating one large contiguous region per request. This reduces external fragmentation and makes it easier to grow or shrink sequences dynamically. The system keeps logical sequence positions separate from physical GPU-memory blocks, similar to virtual memory. The main benefits are better memory utilization, easier request admission, and more efficient sharing or reuse. The trade-off is extra page-table-like metadata and some indirection overhead, but this is usually much smaller than the cost of wasted GPU memory.
Q10. Prefix Cache and Multi-Tenant Security	Prefix caching reuses KV states for repeated prompt prefixes, such as system prompts, common instructions, or shared documents. This can reduce both prefill latency and compute cost. In a multi-tenant system, however, cache keys must include tenant identity, model version, tokenizer state, prompt content, and relevant configuration to avoid accidental cross-tenant reuse. Security and isolation are critical because KV states encode information about the original prompt. I would therefore design cache namespaces, strict access controls, eviction policies, and cryptographic or content-based cache keys carefully.
Q11. Speculative Decoding	Speculative decoding uses a smaller draft model to propose several future tokens, then verifies them in parallel with the larger target model. If many proposed tokens are accepted, the target model can advance by multiple tokens per expensive verification step. The speedup depends on the draft model cost, the acceptance rate, and the verification batch size. A weak draft model gives low acceptance, while an overly expensive draft model erodes the benefit. The ideal setup maximizes accepted tokens per unit of total compute while preserving exactly the target-model distribution.
Q12. INT4 / INT8 / FP8 / FP4 Quantization	Quantization reduces memory footprint and bandwidth and can increase throughput when the hardware has native low-precision support. INT8 is relatively robust and widely supported, while INT4 gives larger memory savings but usually needs more careful scaling and outlier handling. FP8 often works well for training and inference because it preserves floating-point dynamic range better than integer formats. FP4 offers even higher compression but requires stronger hardware and algorithmic support. The real trade-off is not just accuracy versus bits; it includes dequantization overhead, scale granularity, kernel support, bandwidth savings, and end-to-end latency.
Q13. KV Cache Quantization	KV-cache quantization targets inference memory rather than model weights. Since the cache grows linearly with sequence length and batch size, quantizing K and V can dramatically improve long-context capacity and decode throughput. However, KV values are reused at every future decoding step, so quantization error can accumulate in attention computation. Keys are especially sensitive because they affect attention scores, while values affect the aggregated content. I would evaluate per-channel or grouped scaling, asymmetric versus symmetric quantization, mixed precision for K and V, and the impact on perplexity, long-context retrieval quality, and latency.
Q14. TP / PP / DP / EP	Tensor Parallelism splits matrix operations within a layer across GPUs and requires frequent collective communication, usually all-reduce or all-gather. Pipeline Parallelism splits layers across devices and reduces per-layer communication but introduces pipeline bubbles and stage-balancing challenges. Data Parallelism replicates the model and processes different batches independently, which is excellent for training but less useful when one model instance itself does not fit on a device. Expert Parallelism is used in MoE models to distribute experts across GPUs, but it introduces all-to-all token routing. The right strategy depends on model size, batch size, interconnect topology, and the communication-to-compute ratio.
Q15. MoE Routing and Load Balancing	In Mixture-of-Experts models, a router selects a small number of experts for each token, typically top-1 or top-2. This increases model capacity without activating every parameter for every token. The main systems issue is load imbalance: if too many tokens choose the same expert, some GPUs become overloaded while others are underutilized. Training therefore often includes load-balancing losses, capacity limits, or expert routing regularization. At inference time, MoE performance depends heavily on all-to-all communication, expert placement, token locality, and topology-aware routing.
Q16. TTFT / ITL / p99 and Capacity Planning	TTFT, or time to first token, mainly measures queueing plus prefill latency. ITL, or inter-token latency, measures decode responsiveness after generation starts. p99 captures tail latency, which is often more important than average latency for production SLOs. Capacity planning requires modeling arrival rate, sequence-length distributions, concurrency, GPU memory, prefill throughput, decode throughput, and scheduling policy. I would monitor both throughput and latency because maximizing tokens per second can easily violate TTFT or p99 targets.
Q17. How to Debug a Sudden Serving Performance Drop	I would first determine whether the regression is in TTFT, decode latency, throughput, memory usage, or tail latency. Then I would compare traffic mix, sequence lengths, batching efficiency, cache hit rates, GPU utilization, SM occupancy, HBM bandwidth, kernel times, NCCL communication, and memory fragmentation before and after the regression. I would also check for software changes such as model version, quantization, compiler flags, kernel selection, or scheduler configuration. The key is to localize the regression by layer: request queue, scheduler, runtime, GPU kernels, memory, or networking, rather than immediately tuning the model.
Q18. CUDA/Triton Kernel Optimization	My first steps are to identify the bottleneck using profiling, then reason about memory coalescing, arithmetic intensity, occupancy, register usage, and synchronization. Tiling improves data reuse by keeping frequently used data in shared memory or registers. Fusion reduces kernel-launch overhead and avoids unnecessary round trips to HBM. Register spilling is a warning sign because it moves values from registers to local memory, which is backed by much slower global memory. In Triton or CUDA, I would tune block sizes, warps, memory layout, vectorization, and fusion based on the actual hardware bottleneck rather than optimizing FLOPs in isolation.
Q19. Prefill–Decode Disaggregation	Prefill and decode have very different resource profiles, so separating them onto different worker pools can improve efficiency. Prefill workers can be optimized for large compute-heavy batches, while decode workers can be optimized for memory bandwidth and low latency. The challenge is transferring KV states between the two stages without making communication the new bottleneck. This architecture also complicates scheduling, fault tolerance, admission control, and GPU resource allocation. It is attractive at scale when the workload is large enough that specialization outweighs KV-transfer and orchestration overhead.
Q20. Design a 70B Long-Context Production Inference Platform	I would start from SLOs, workload distribution, maximum context length, concurrency, and quality requirements. For a 70B model, I would likely use tensor parallelism within a node, possibly pipeline or expert parallelism depending on the architecture, and GQA plus paged KV cache to control KV-memory growth. I would use FlashAttention for prefill, continuous batching for high utilization, prefix caching where prompts repeat, speculative decoding if acceptance rate is favorable, and weight/KV quantization if quality allows. For long context, I would explicitly model KV-cache capacity, HBM bandwidth, TTFT, ITL, and interconnect traffic. At larger scale, I would consider prefill–decode disaggregation, topology-aware placement, admission control, SLO-aware scheduling, and detailed observability for GPU kernels, memory, cache hit rates, queueing, and p99 latency.