SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding calibrates pruning on model-generated tokens and adds a decoding-specific sparse kernel. We examine the conditions behind its maximum 1.48× speedup on A100…