FastDeploy

FastDeploy is an inference deployment toolkit for large language models and vision-language models based on PaddlePaddle, designed to provide out-of-the-box production-grade deployment solutions. This tool supports various mainstream hardware platforms and integrates load-balanced PD separation, unified KV cache transmission, and multiple advanced acceleration technologies. Developers can achieve rapid deployment through OpenAI API-compatible interfaces and optimize inference performance using full quantization format support.

FastDeploy is a production-grade inference deployment toolkit designed for Large Language Models (LLMs) and Vision-Language Models (VLMs), built upon the PaddlePaddle ecosystem. This project aims to address the complexity of deploying models across multi-hardware environments by providing load-balanced PD disaggregation, unified KV cache transmission, and various advanced acceleration technologies, significantly improving inference throughput and resource utilization. Its core design philosophy lies in compatibility with mainstream ecosystems (such as vLLM interface compatibility) and providing extensive support for domestic and mainstream hardware, thereby lowering the technical barrier for enterprise-level model implementation.