Supercharge Your LLM with the Fastest KV Cache Layer

A Golang implemented Redis Server and Cluster. Go 语言实现的 Redis 服务器和分布式集群

Unified KV Cache Compression Methods for Auto-Regressive Models

LLM KV cache compression made easy

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.

Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025)

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Awesome-LLM-KV-Cache: A curated list of 📙Awesome LLM KV Cache Papers with Codes.

[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

[Survey] Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

LLM inference with 7x longer context. Pure C, zero dependencies. Lossless KV cache compression + single-header library.

HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of HierarchicalKV is to store key-value feature-embeddings on high-bandwidth memory (HBM) of GPUs and in host memory. It also can be used as a generic key-value storage.

Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024)

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, MoE expert parallelism, OpenAI-compatible serving

[NeurIPS'25] KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems

Alibaba Cloud's high-performance KVCache system for LLM inference, with components for global cache management, inference simulation(HiSim), and more.

Completion After Prompt Probability. Make your LLM make a choice

TurboQuant KV cache compression for MLX with fused Metal kernels. 4.6x compression at 98% FP16 speed.