Tiiny-AI/PowerInfer
View on GitHub ↗ ★ 9,874 C++
High-speed Large Language Model Serving for Local Deployment
6 mentions on Hacker News by 6 people: 2 stories, 4 comments · first 2023-12-19 · latest 2024-06-12
20082026
-
Project Github link: https://github.com/SJTU-IPADS/PowerInfer
comment by helloericsf · 2024-06-12 · in 21.2× faster than llama.cpp? plus 40% memory usage reduction -
this sounds related to this: https://arxiv.org/abs/2312.12456 https://github.com/SJTU-IPADS/PowerInfer
comment by toisanji · 2024-04-17 · in Show HN: Speeding up LLM inference 2x times (possibly) -
▲ 417
High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUsstory by dataminer · 2023-12-20 · 83 comments
-
…neurons to the GPU and leaving "cold" ones on the CPU) is a neat one on the surface. [0] https://github.com/SJTU-IPADS/PowerInfer
comment by girvo · 2023-12-19 · in PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf] -
> PowerInfer’s source code is publicly available at https://github.com/SJTU-IPADS/PowerInfer --- Just curious - PowerInfer seems to market itself by running very large models (40B, 70B) on something like a 4090. If I have, say, a 3060…
comment by LoganDark · 2023-12-19 · in PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf] -
▲ 4
PowerInfer: High-Speed Large Language Model Serving on Consumer-Grade GPUsstory by limoce · 2023-12-19 · 1 comment