Inference

The Willow Inference Server allows users to self-host high-speed language inference tasks for various applications. It supports a range of functionalities including speech-to-text, text-to-speech, and large language model processing. Users can access official documentation and community discussions to optimize their experience with the platform.

ncnn is a high-performance neural network forward computation framework deeply optimized for mobile platforms. The framework has no third-party dependencies and features cross-platform capabilities, outperforming all known open-source frameworks on mobile CPUs. Developers can easily port deep learning models to mobile devices using ncnn to build various intelligent applications.

ncnn is a high-performance neural network forward computation framework specifically optimized for mobile platforms, designed to simplify the deployment of deep learning algorithms on mobile devices. The framework has no third-party dependencies and features cross-platform capabilities, with execution speeds on mobile CPUs that outperform all currently known open-source frameworks. Currently, ncnn is widely used in various mainstream applications under Tencent, helping developers easily build intelligent applications.

RTP-LLM is a high-performance large language model inference acceleration engine developed by the Alibaba Foundation Model Inference team. This engine has been widely applied in various Alibaba business scenarios such as Taobao and Tmall, and supports multiple mainstream model formats and hardware backends. By integrating advanced operator optimization, quantization techniques, and distributed inference capabilities, it provides developers with efficient and flexible deployment solutions.

This repository provides a curated list of LLM API providers that offer permanent free tiers for text inference. It categorizes services into direct provider APIs and third-party inference platforms, detailing model capabilities, context windows, and rate limits for each. The collection serves as a resource for developers seeking cost-effective access to various language models without requiring credit card information.

FastDeploy is an inference deployment toolkit for large language models and vision-language models based on PaddlePaddle, designed to provide out-of-the-box production-grade deployment solutions. The tool supports various mainstream hardware platforms and integrates core technologies such as load-balanced PD separation, unified KV cache transmission, and full quantization format support. Developers can quickly achieve high-performance model inference and deployment through its OpenAI API-compatible service interface.

Tair KVCache is an Alibaba Cloud system designed to accelerate Large Language Model inference through distributed memory pooling and dynamic multi-level caching. The project provides a centralized manager for unified metadata handling and a simulation tool for predicting performance metrics without requiring GPU resources. These components work together to improve inference efficiency while reducing overall infrastructure costs.

FastDeploy is an inference deployment toolkit for large language models and vision-language models based on PaddlePaddle, aiming to provide out-of-the-box production-grade deployment solutions. The toolkit supports various mainstream hardware platforms and integrates core technologies such as load-balanced PD separation, unified KV cache transmission, and full quantization format support. By being compatible with OpenAI API and vLLM interfaces, it helps developers efficiently implement model inference and online service deployment.

RTP-LLM is a high-performance large model inference acceleration engine developed by the Alibaba Foundation Model Inference Team, widely used in various business scenarios such as Taobao and Tmall. By integrating various advanced CUDA kernels and quantization techniques, the engine significantly improves model inference performance and efficiency. Furthermore, it possesses high flexibility, supporting multiple model formats, multimodal inputs, and LoRA service deployment.

vLLM Kunlun is a community-maintained hardware plugin that enables the seamless execution of vLLM on Kunlun XPU devices. It functions as a hardware-pluggable interface, allowing users to run various large language and multimodal models without modifying the original vLLM source code. The project supports advanced features like quantization, LoRA fine-tuning, and hardware-accelerated graph optimization to ensure high-performance inference.

LiteRT-LM is a high-performance, production-ready inference framework designed by Google for deploying Large Language Models on edge devices. It supports a wide range of platforms including Android, iOS, desktop, and IoT, while leveraging GPU and NPU hardware acceleration for optimal performance. The framework enables advanced capabilities such as multi-modality and function calling, powering on-device AI experiences in various Google products.