This repository is a fork of Tiiny-AI/PowerInfer for our UCF CAP 6614 project work. For full upstream build instructions, model support, and engine documentation, see README-PowerInfer.md in this repo, and the original projects linked below.
| Name | |
|---|---|
| Cuong Dang | cuong.dang@ucf.edu |
| Hai Nguyen | hai.nguyen@ucf.edu |
| Joshua Lowe | jo201760@ucf.edu |
| Lam Nguyen | lam.nguyen@ucf.edu |
| Title | PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU |
| Authors | Xue, Y., et al. (SJTU IPADS) |
| Venue | SOSP 2024 |
Short description: PowerInfer exploits locality in LLM inference: for any given input, only a small set of neurons are hot (frequently activated). The system pre-loads hot neurons on the GPU while computing cold neurons on the CPU, reducing memory pressure and data movement. Reported results include up to about 11× speedup over llama.cpp on a single NVIDIA RTX 4090 for large models (e.g. 175B-class).
Upstream code (reference): github.com/SJTU-IPADS/PowerInfer
We deploy PowerInfer on a consumer GPU with an OPT or LLaMA-family model, then:
- Profile the hot/cold neuron distribution across layers and runs.
- Compare end-to-end inference speed against llama.cpp and vLLM under controlled settings.
- Analyze how the hot-neuron set varies with different input types (e.g. topic, length, or task format).
Project artifacts (scripts, plots, and result tables) live under Reports/ in this repository.
Full procedure (prerequisites, UCF Newton, Apple rebuild options, and Phases 1–4) is in Reports/ExperimentalSetup.md. Guidelines in brief:
- Where it runs best: The paper’s GPU–CPU hybrid (hot on GPU, cold on CPU) is built for NVIDIA CUDA. On Apple Silicon, runs are CPU-oriented (e.g. NEON/BLAS, not the discrete-GPU hot/cold path). GPU benchmarks and apples-to-apples comparisons with that design belong on a CUDA machine—this project uses UCF Newton; profiling and much analysis can be done locally.
- ~16 GB VRAM nodes: A single F16 dense run is most realistic for ~7B; 13B/70B often need a larger-GPU partition, quantized baselines (e.g. llama.cpp), or you document partial-GPU/CPU behavior. Phase 2 neuron profiling is CPU and disk only, not a VRAM bottleneck.
- HPC env (Newton): Typical bootstrap is
module loadan Anaconda module, a project.venv, andpip install -r requirements.txt—see the report for exact module names and run scripts underReports/scripts/ucf_newton/.
| Resource | URL |
|---|---|
| This fork (base) | Tiiny-AI/PowerInfer |
| Original PowerInfer (SJTU) | SJTU-IPADS/PowerInfer |
This project inherits the upstream project’s terms; see the license files in the tree and README-PowerInfer.md for full notices. When you cite the system in academic work, use the SOSP 2024 PowerInfer paper and the upstream repository as appropriate to your style guide.