Hacker GitHub

← All repos

mlc-ai/mlc-llm

View on GitHub ↗ ★ 23,293 Python

Universal LLM Deployment Engine with ML Compilation

20 mentions on Hacker News by 15 people: 3 stories, 17 comments · first 2023-04-29 · latest 2025-09-08

Mentions per year 20 mentions · busiest year 2023 (17)
20082026
AllStoriesComments NewestTopOldest
  1. Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm
    comment by SparkyMcUnicorn · 2025-09-08 · in Experimenting with Local LLMs on macOS
  2. …need to define/compile models for it with TVM. Here is the list of supported model architectures: https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/m... llama.cpp has a much bigger supported model list, as does vLLM and of course…
    comment by lhl · 2024-12-24 · in Making AMD GPUs competitive for LLM inference (2023)
  3. there's a pretty cool Vulkan LLM engine here for example: https://github.com/mlc-ai/mlc-llm
    comment by 0xDEADFED5 · 2024-02-13 · in AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source
  4. I got this project[0] running on a Pixel. Looks like it works on some iPhones/iPads as well. [0] https://github.com/mlc-ai/mlc-llm
    comment by SparkyMcUnicorn · 2023-11-13 · in Fast and Portable Llama2 Inference on the Heterogeneous Edge
  5. …Details: - Blog post: https://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infer... - Project: https://github.com/mlc-ai/mlc-llm
    comment by junrushao1994 · 2023-10-20 · in Scaling LLama2-70B with Multiple Nvidia/AMD GPU
  6. …a shoutout to MLC LLM, which runs LLM models on basically any API that's widely available: https://github.com/mlc-ai/mlc-llm
    comment by tormeh · 2023-10-07 · in AMD may get across the CUDA moat
  7. …docs back in, but also here's a step-by-step w/ my adventures getting TVM/MLC working w/ an APU: https://github.com/mlc-ai/mlc-llm/issues/787 From my experience, ROCm is improving, but there's a good reason that Nvidia has 90% market share…
    comment by lhl · 2023-09-27 · in ROCm is AMD's priority, executive says
  8. Maybe they're talking about https://github.com/mlc-ai/mlc-llm which is used for web-llm ( https://github.com/mlc-ai/web-llm )? Seems to be using TVM.
    comment by capableweb · 2023-09-26 · in Ollama for Linux – Run LLMs on Linux with GPU Acceleration
  9. …or https://octoml.ai/blog/using-swift-and-apache-tvm-to-develop... or https://github.com/mlc-ai/mlc-llm
    comment by homarp · 2023-09-12 · in Fine-tune your own Llama 2 to replace GPT-3.5/4
  10. Another Engine: - https://github.com/mlc-ai/mlc-llm
    comment by kordlessagain · 2023-08-14 · in Ask HN: Are you training and running custom LLMs and how are you doing it?
  11. Things are already possible on today's hardware, see https://github.com/mlc-ai/mlc-llm which allows many models to be run on M1/M2 Macs, WASM, iOS and more. The main limiting factor will be small enough, high quality enough models that…
    comment by jroesch · 2023-07-19 · in Apple Tests ‘Apple GPT,’ Develops Generative AI Tools to Catch OpenAI
  12. With any luck projects like MLC will help close the gap https://github.com/mlc-ai/mlc-llm
    comment by lachlan_gray · 2023-06-13 · in Llama.cpp: Full CUDA GPU Acceleration
  13. …Some other runtimes like Apache TVM also have their own quant implementations: https://github.com/mlc-ai/mlc-llm For training, 4-bit bitsandbytes is SOTA, as far as I know: https://huggingface.co/blog/4bit-transformers-bitsandbytes TBH I
    comment by brucethemoose2 · 2023-06-10 · in LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)
  14. …know... Hippo is closed source for now. Its comparable to Apache TVM's vulkan in speed on cuda, see https://github.com/mlc-ai/mlc-llm But honestly, the biggest advantage of llama.cpp for me is being able to split a model so performantly.…
    comment by brucethemoose2 · 2023-06-06 · in GGML – AI at the Edge
  15. https://github.com/mlc-ai/mlc-llm
    comment by TMWNN · 2023-06-02 · in 27 years later and the Psion 3a is still wonderful (2020)
  16. ▲ 2
    Bringing Hardware Accelerated Language Models to Android Devices
    story by crowwork · 2023-05-09 · 0 comments
  17. …(investigate how many performance cores you've got). GPU is slower, but more energy efficient. https://github.com/mlc-ai/mlc-llm gives me way better GPU performance compared to oobabooga, but they only support a couple of models right now,…
    comment by execveat · 2023-05-08 · in RasaGPT: First headless LLM chatbot built on top of Rasa, Langchain and FastAPI
  18. …With CUDA, I only get about 1-3 tokens per second. Are these the only supported models as of now? https://github.com/mlc-ai/mlc-llm/blob/d3e7f16c54238b7da5e78...
    comment by syntaxing · 2023-04-30 · in MLC-LLM: GPT/Llama on consumer-class GPUs and phones
  19. ▲ 303
    MLC-LLM: GPT/Llama on consumer-class GPUs and phones
    story by junrushao1994 · 2023-04-30 · 106 comments
  20. ▲ 3
    MLC LLM: Universal LLM Deployment with GPU Acceleration
    story by crowwork · 2023-04-29 · 1 comment