Memory Bandwidth as the Primary Inference Bottleneck
Autoregressive transformer decoding is fundamentally memory-bandwidth bound rather than compute bound. Every generated token requires transferring model weights and key-value state from High Bandwidth Memory into SRAM registers. Autoregressive transformer decoding is fundamentally memory-bandwidth bound rather than compute bound. Every generated token requires transferring model weights and key-value state from High Bandwidth Memory into SRAM registers.Autoregressive transformer decoding is fundamentally memory-bandwidth bound rather than compute bound. Every generated token requires transferring model weights and key-value state from High Bandwidth Memory into SRAM registers.Autoregressive transformer decoding is fundamentally memory-bandwidth bound rather than compute bound. Every generated token requires transferring model weights and key-value state from High Bandwidth Memory into SRAM registers.Autoregressive transformer decoding is fundamentally memory-bandwidth bound rather than compute bound. Every generated token requires transferring model weights and key-value state from High Bandwidth Memory into SRAM registers.
Speculative Execution with Lightweight Draft Models
By pairing a compact draft model with a target verifier pass, speculative decoding verifies multiple candidate tokens in a single batched forward pass, guaranteeing mathematically identical output distributions. By pairing a compact draft model with a target verifier pass, speculative decoding verifies multiple candidate tokens in a single batched forward pass, guaranteeing mathematically identical output distributions.By pairing a compact draft model with a target verifier pass, speculative decoding verifies multiple candidate tokens in a single batched forward pass, guaranteeing mathematically identical output distributions.By pairing a compact draft model with a target verifier pass, speculative decoding verifies multiple candidate tokens in a single batched forward pass, guaranteeing mathematically identical output distributions.By pairing a compact draft model with a target verifier pass, speculative decoding verifies multiple candidate tokens in a single batched forward pass, guaranteeing mathematically identical output distributions.
Production Deployment Trade-Offs and Tail Latency
Under high concurrency, speculative acceptance rates must be balanced against arithmetic intensity. Disaggregated prefill-decode scheduling isolates compute-heavy prompt ingestion from latency-sensitive token streaming. Under high concurrency, speculative acceptance rates must be balanced against arithmetic intensity. Disaggregated prefill-decode scheduling isolates compute-heavy prompt ingestion from latency-sensitive token streaming.Under high concurrency, speculative acceptance rates must be balanced against arithmetic intensity. Disaggregated prefill-decode scheduling isolates compute-heavy prompt ingestion from latency-sensitive token streaming.Under high concurrency, speculative acceptance rates must be balanced against arithmetic intensity. Disaggregated prefill-decode scheduling isolates compute-heavy prompt ingestion from latency-sensitive token streaming.Under high concurrency, speculative acceptance rates must be balanced against arithmetic intensity. Disaggregated prefill-decode scheduling isolates compute-heavy prompt ingestion from latency-sensitive token streaming.
def expected_speedup(acceptance_rate: float, gamma: int, cost_ratio: float) -> float:
numerator = 1.0 - (acceptance_rate ** (gamma + 1))
denominator = (1.0 - acceptance_rate) * (1.0 + gamma * cost_ratio)
return round(numerator / denominator, 3)