Const-me/Cgml on Hacker News

…tech is called nanite virtualized geometry). In practice they work fine on all GPUs, ML included: https://github.com/Const-me/Cgml

comment by Const-me · 2024-12-24 · in Making AMD GPUs competitive for LLM inference (2023)

…quantization which works on all GPUs, no hardware support necessary, just simple HLSL codes: https://github.com/Const-me/Cgml?tab=readme-ov-file#bcml1-co...

comment by Const-me · 2024-12-24 · in Making AMD GPUs competitive for LLM inference (2023)

…ML inference to alternative GPU APIs. I did it for D3D11 and the performance is pretty good too: https://github.com/Const-me/Cgml The only catch is, for some reason developers of ML libraries like PyTorch aren’t interested in open GPU APIs…

comment by Const-me · 2024-12-24 · in Making AMD GPUs competitive for LLM inference (2023)

I wonder how does the perf in tokens/second compares to my version of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... BTW, see that section of the readme about quantization: https://github.com/Const-me/Cgml/tree/mas…

comment by Const-me · 2024-12-15 · in Fast LLM Inference From Scratch (using CUDA)

…for a Direct3D 11.0 compute shader. Here’s the fastest version I managed to make back then: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... As you see, I have implemented 32×32 tiling, using thread groups of 32×8 threads,…

comment by Const-me · 2024-11-11 · in Optimizing a WebGPU Matmul Kernel for 1 TFLOP

…support ROCm. BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs…

comment by Const-me · 2024-10-02 · in AMD GPU Inference

…For Mistral I decided to rely on C# as much as possible and I like the result much better https://github.com/Const-me/Cgml/

comment by Const-me · 2024-08-12 · in Ask HN: What are some "toy" projects you used to learn neural networks hands-on?

…of a relatively complicated COM interface implemented on top of ComLightInterop. C# interface: https://github.com/Const-me/Cgml/blob/master/CGML/CgmlNet/iC... C++ interface: https://github.com/Const-me/Cgml/blob/master/CGML/Cgml/API/i...…

comment by Const-me · 2024-07-02 · in Writing GUI apps for Windows is painful

…year or so I have implemented inference of Whisper https://github.com/Const-me/Whisper and Mistral https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... models on all GPUs which support Direct3D 11.0 API. The performance is IMO…

comment by Const-me · 2024-05-20 · in Llama3 implemented from scratch

…same applies to ML libraries, most folks are using Python/Torch/CUDA stack. For that hobby project https://github.com/Const-me/Cgml/ I had to re-implement the entire tech stack in C#/C++/HLSL.

comment by Const-me · 2024-04-30 · in Asynchronous Programming in C#

…just shipping the LLM with game right? That might be a nice application for this library of mine: https://github.com/Const-me/Cgml/ That’s an open source Mistral ML model implementation which runs on GPUs (all of them, not just nVidia),…

comment by Const-me · 2024-04-08 · in Groq CEO: 'We No Longer Sell Hardware'

…a similar project with C# for the higher-level pieces, and Direct3D 11 GPU API instead of Vulkan: https://github.com/Const-me/Cgml > don't see many other people using Vulkan for GPGPU, and there may be many good reasons for that I think…

comment by Const-me · 2024-03-17 · in Show HN: Richard – A CNN written in C++ and Vulkan (no ML or math libs)

If you just want to run Mistral on Windows, you could try my port: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... The setup is relatively easy: install .NET runtime, download 4.5 GB model file from BitTorrent, unpack a…

comment by Const-me · 2024-03-17 · in Ask HN: How to get started with local language models?

…sampling during inference, most ML models are doing it rather inefficiently. Here’s a better way: https://github.com/Const-me/Cgml/blob/master/Readme.md#rando... My HLSL is easily portable to CUDA, which has `__syncthreads` and `atomicInc`…

comment by Const-me · 2024-02-22 · in Unexpected responses from ChatGPT: Incident Report

You could try my Mistral implementation: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral...

comment by Const-me · 2024-02-13 · in Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC

I did a few times with Direct3D 11 compute shaders. Here’s an open-source example: https://github.com/Const-me/Cgml Pretty sure Vulkan gonna work equally well, at the very least there’s an open source DXVK project which implements D3D11 on…

comment by Const-me · 2024-02-13 · in AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

Here’s an example of a custom 4 bits/weight codec for ML weights: https://github.com/Const-me/Cgml/blob/master/Readme.md#bcml1... llama.cpp does it slightly differently but still, AFAIK their quantized data formats are conceptually similar…

comment by Const-me · 2024-01-27 · in Brave Leo now uses Mixtral 8x7B as default

…GPU from any vendor with at least 6GB VRAM, you could test my implementation of Mistral model: https://github.com/Const-me/Cgml Screenshot: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral...

comment by Const-me · 2024-01-20 · in Efficient LLM inference solution on Intel GPU

…see that library which implements ML-targeted compute shaders for C# with minimal friction: https://github.com/Const-me/Cgml The backend implemented in C++ is rather simple, just binds resources and dispatches these shaders. I think the…

comment by Const-me · 2024-01-10 · in Vcc – The Vulkan Clang Compiler

Minor update https://github.com/Const-me/Cgml/releases/tag/1.1a Can’t edit that comment anymore, too late.

comment by Const-me · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

Is that explanation better? https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... Same Mistral Instruct 0.2 model, different implementation.

comment by Const-me · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

The model is on BitTorrent, see readme for the frontend app: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... The torrent file is also inside the MistralChat.zip archive.

comment by Const-me · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

I made a free and open source Windows equivalent: https://github.com/Const-me/Cgml/releases/tag/1.1

comment by Const-me · 2024-01-08 · in I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

For implementing stuff from scratch, if you use Windows you could try my C# based library for that: https://github.com/Const-me/Cgml/ It’s vendor agnostic, so HLSL instead of CUDA or Triton. Here’s the compute shaders implementing…

comment by Const-me · 2024-01-04 · in Ask HN: Best way to learn GPU programming?

…I’d like to note that my GPU-only implementation of Mistral uses slightly over 5GB of VRAM: https://github.com/Const-me/Cgml Runs pretty good on most consumer-grade GPUs, but so far it only supports Windows OS.

comment by Const-me · 2023-12-20 · in High-Speed Large Language Model Serving on PCs with Consumer-Grade GPUs

…really, that code only runs while importing the PyTorch format. See readme for the frontend app: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... When loading the model from *.cgml, the model file already contains compressed…

comment by Const-me · 2023-12-17 · in AMD's CDNA 3 Compute Architecture

…a better technology stack. I have ported Whisper https://github.com/Const-me/Whisper/ and Mistral https://github.com/Const-me/Cgml/ to D3D11. I don’t remember how much time I spent, but given both were unpaid part-time hobby projects,…

comment by Const-me · 2023-12-17 · in AMD's CDNA 3 Compute Architecture

Windows equivalent: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Runs on GPUs, uses about 5GB VRAM. On integrated GPUs generates 1-2 tokens/second, on discrete ones often over 20 tokens/second.

comment by Const-me · 2023-12-17 · in Run Mistral 7B on M1 Mac

…Matrix multiplication compute shader which uses group shared memory for cooperative loads: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... > tensor cores When running inference on end-user computers, for many practical…

comment by Const-me · 2023-12-15 · in Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

I have recently published an AI-related open-source project entirely based on compute shaders https://github.com/Const-me/Cgml and I’m super happy with the workflow. Possible to implement very complicated things without compiling a single…

comment by Const-me · 2023-12-14 · in Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'

…I wonder why Microsoft helps nVidia, instead of using their own technology? Here’s an example: https://github.com/Const-me/Cgml

comment by Const-me · 2023-12-14 · in Windows AI Studio Preview

…an offline chat program to interact with that model. Here’s a guide for the offline chat program: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral...

comment by Const-me · 2023-12-12 · in Show HN: Cgml, GPU-targeted vendor-agnostic AI library for Windows

▲ 2

Show HN: Cgml, GPU-targeted vendor-agnostic AI library for Windows

story by Const-me · 2023-12-12 · 1 comment