Hacker GitHub

← All repos

algorithmicsuperintelligence/optillm

View on GitHub ↗ ★ 4,334 Python

Optimizing inference proxy for LLMs

22 mentions on Hacker News by 3 people: 9 stories, 13 comments · first 2024-09-13 · latest 2025-06-19

Mentions per year 22 mentions · busiest year 2025 (12)
20082026
AllStoriesComments NewestTopOldest
  1. …Minimum success rate for a strategy to be used during inference The configs are all defined here - https://github.com/codelion/optillm/blob/main/optillm/plugin...
    comment by codelion · 2025-06-02 · in Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
  2. …phase where we refine and merge existing strategies. The experiments were run with this config - https://github.com/codelion/optillm/blob/main/optillm/plugin... which allows a maximum of 10 strategies of each type.
    comment by codelion · 2025-06-02 · in Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
  3. We have some examples in the plugin README: https://github.com/codelion/optillm/tree/main/optillm/plugin... E.g. This was the strategy discovered by optiLLM for solving word problems: *Refined Strategy for Solving Word Problems:* 1.…
    comment by codelion · 2025-06-02 · in Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
  4. Not standard but one of several techniques, you can see them in our open source inference proxy - https://github.com/codelion/optillm Cerebras has used optillm for optimising inference with techniques like CePO and LongCePO.
    comment by codelion · 2025-04-28 · in Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
  5. …to improve the performance of local LLMs with inference time techniques. You can try with optillm - https://github.com/codelion/optillm it is possible to match the performance of larger models on narrow tasks by doing more at inference.
    comment by codelion · 2025-04-21 · in Local LLM inference – impressive but too hard to work with
  6. …use cases, like optimizing inference, can be a good way to learn. On that note, you might find https://github.com/codelion/optillm useful – it optimizes LLM inference and could give you practical experience with GPU utilization. What kind…
    comment by codelion · 2025-02-21 · in Introduction to CUDA programming for Python developers
  7. …this be used to implement a sampler based on entropy like entropix (implemented in optillm here - https://github.com/codelion/optillm/blob/main/optillm/entrop... )
    comment by codelion · 2025-02-03 · in Show HN: Klarity – Open-source tool to analyze uncertainty/entropy in LLM output
  8. …you can use the eval script I have in optillm to benchmark on it - https://github.com/codelion/optillm/blob/main/scripts/eval_a...
    comment by codelion · 2025-01-21 · in Kimi K1.5: Scaling Reinforcement Learning with LLMs
  9. Optillm - https://github.com/codelion/optillm optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is…
    comment by codelion · 2024-10-28 · in Ask HN: What are you working on? (October 2024)
  10. …it will last - the end game surely will be a trainable sampler. But for now - enjoy tinkering: https://github.com/codelion/optillm implements a few of these techniques optillm authors suggest that the additional computations in Entropics…
    comment by zby · 2024-10-26 · in Detecting when LLMs are uncertain
  11. Can it be mixed with the sampling based approaches from optillm ( https://github.com/codelion/optillm )?
    comment by zby · 2024-10-22 · in Show HN: Steiner – An open-source reasoning model inspired by OpenAI o1
  12. This is good I also had worked on something similar in optillm - https://github.com/codelion/optillm . You can do this with any LLM and several optimization techniques (including cot_reflection) like mcts, plansearch, moa etc.
    comment by codelion · 2024-09-16 · in g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains
  13. …spent some time on 2) and implemented several approaches in this open source optimising llm proxy - https://github.com/codelion/optillm In my experience it does work quite well, but we probably need different techniques for different tasks.
    comment by codelion · 2024-09-13 · in Notes on OpenAI's new o1 chain-of-thought models