FMInference/FlexLLMGen
View on GitHub ↗ ★ 9,343 Python archived
Running large language models on a single GPU for throughput-oriented scenarios.
14 mentions on Hacker News by 13 people: 2 stories, 12 comments · first 2023-02-20 · latest 2023-12-03
20082026
-
See https://github.com/FMInference/FlexGen This is from much earlier this year and seems to be the same idea, only loading part of the model into the GPU at a time.
comment by andy99 · 2023-12-03 · in Run 70B LLM Inference on a Single 4GB GPU with This New Technique -
…make a took like yours? This was mee a few weeks ago. I got interested in all this when FlexGen ( https://github.com/FMInference/FlexGen ) was announced, which allowed to run inference using OPT model on consumer hardware. I'm an avid user…
comment by lxe · 2023-03-22 · in Show HN: Finetune LLaMA-7B on commodity GPUs using your own text -
…I only know about these: - use 8bit precision - https://github.com/bigscience-workshop/petals - https://github.com/FMInference/FlexGen - https://github.com/microsoft/DeepSpeed Anything that could bring this to a 10GB 3080 or 24GB 3090…
comment by underlines · 2023-03-03 · in Facebook LLAMA is being openly distributed via torrents -
…a similar set of models (and the code has support for even swapping data out to disk if needed): https://github.com/FMInference/FlexGen And here are some benchmarks running OPT-175B purely on (a very beefy) CPU machine. Note that the…
comment by gpm · 2023-03-03 · in Facebook LLAMA is being openly distributed via torrents -
Try this, it's for running llms that won't fit in the gpu: https://github.com/FMInference/FlexGen
comment by version_five · 2023-03-03 · in Facebook LLAMA is being openly distributed via torrents -
…Fortunately people have started writing implementations for pipelining across multiple gpus. https://github.com/Ying1123/FlexGen
comment by nullc · 2023-03-03 · in Replika users fell in love with their AI chatbot companions. Then they lost them -
…out of it, but it probably would be good enough to run your own ChatGPT or Copilot locally. [1]: https://github.com/FMInference/FlexGen
comment by ddren · 2023-02-27 · in Open source implementation for LLaMA-based ChatGPT -
…FSF like players that are motivated by freedom. [0] https://github.com/LAION-AI/Open-Assistant [1] https://github.com/FMInference/FlexGen
comment by version_five · 2023-02-25 · in Data Science and LLM (LLaMA vs. GPT-3): A Revolution That Will Change the World -
If you're patient, https://github.com/FMInference/FlexGen lets you trade off GPU RAM for system RAM or even disk space.
comment by sp332 · 2023-02-25 · in LLaMA: A foundational, 65B-parameter large language model -
…run on consumer hardware, based on the methodology used for LLaMA (and other similar papers e.g. https://github.com/FMInference/FlexGen ), at some point in the next 6-24 months.
comment by simonw · 2023-02-24 · in LLaMA: A foundational, 65B-parameter large language model -
…reach of the average user. [1] https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b... [2] https://github.com/FMInference/FlexGen [3] https://ai.googleblog.com/2022/12/accelerating-text-generati...
comment by moyix · 2023-02-21 · in Whispers of A.I.’s Modular Future -
It would be helpful to upload the paper to Arxiv, for better accessibility and visibility. https://github.com/Ying1123/FlexGen/blob/main/docs/paper.pdf https://docs.google.com/viewer?url=https://github.com/Ying11...
comment by albertzeyer · 2023-02-20 · in Running large language models like ChatGPT on a single GPU