Hacker GitHub

← All repos

algorithmicsuperintelligence/optillm

View on GitHub ↗ ★ 4,334 Python

Optimizing inference proxy for LLMs

22 mentions on Hacker News by 3 people: 9 stories, 13 comments · first 2024-09-13 · latest 2025-06-19

Mentions per year 22 mentions · busiest year 2025 (12)
20082026
AllStoriesComments NewestTopOldest
  1. ▲ 4
    Show HN: DeepThink Plugin – Bring Gemini 2.5's parallel reasoning to open models
    story by codelion · 2025-06-19 · 0 comments
  2. …Minimum success rate for a strategy to be used during inference The configs are all defined here - https://github.com/codelion/optillm/blob/main/optillm/plugin...
    comment by codelion · 2025-06-02 · in Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
  3. …phase where we refine and merge existing strategies. The experiments were run with this config - https://github.com/codelion/optillm/blob/main/optillm/plugin... which allows a maximum of 10 strategies of each type.
    comment by codelion · 2025-06-02 · in Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
  4. We have some examples in the plugin README: https://github.com/codelion/optillm/tree/main/optillm/plugin... E.g. This was the strategy discovered by optiLLM for solving word problems: *Refined Strategy for Solving Word Problems:* 1.…
    comment by codelion · 2025-06-02 · in Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
  5. ▲ 48
    Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
    story by codelion · 2025-06-02 · 13 comments
  6. ▲ 397
    Show HN: AutoThink – Boosts local LLM performance with adaptive reasoning
    story by codelion · 2025-05-28 · 68 comments
  7. Not standard but one of several techniques, you can see them in our open source inference proxy - https://github.com/codelion/optillm Cerebras has used optillm for optimising inference with techniques like CePO and LongCePO.
    comment by codelion · 2025-04-28 · in Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
  8. …to improve the performance of local LLMs with inference time techniques. You can try with optillm - https://github.com/codelion/optillm it is possible to match the performance of larger models on narrow tasks by doing more at inference.
    comment by codelion · 2025-04-21 · in Local LLM inference – impressive but too hard to work with
  9. …use cases, like optimizing inference, can be a good way to learn. On that note, you might find https://github.com/codelion/optillm useful – it optimizes LLM inference and could give you practical experience with GPU utilization. What kind…
    comment by codelion · 2025-02-21 · in Introduction to CUDA programming for Python developers
  10. ▲ 2
    Show HN: OptiLLMBench – Test how inference optimization tricks scale up LLMs
    story by codelion · 2025-02-17 · 0 comments
  11. …this be used to implement a sampler based on entropy like entropix (implemented in optillm here - https://github.com/codelion/optillm/blob/main/optillm/entrop... )
    comment by codelion · 2025-02-03 · in Show HN: Klarity – Open-source tool to analyze uncertainty/entropy in LLM output
  12. …you can use the eval script I have in optillm to benchmark on it - https://github.com/codelion/optillm/blob/main/scripts/eval_a...
    comment by codelion · 2025-01-21 · in Kimi K1.5: Scaling Reinforcement Learning with LLMs
  13. ▲ 67
    Launch HN: Patched (YC S24) – AI workflows for post-code tasks
    story by rohansood15 · 2024-10-31 · 20 comments
  14. Optillm - https://github.com/codelion/optillm optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is…
    comment by codelion · 2024-10-28 · in Ask HN: What are you working on? (October 2024)
  15. …it will last - the end game surely will be a trainable sampler. But for now - enjoy tinkering: https://github.com/codelion/optillm implements a few of these techniques optillm authors suggest that the additional computations in Entropics…
    comment by zby · 2024-10-26 · in Detecting when LLMs are uncertain
  16. Can it be mixed with the sampling based approaches from optillm ( https://github.com/codelion/optillm )?
    comment by zby · 2024-10-22 · in Show HN: Steiner – An open-source reasoning model inspired by OpenAI o1
  17. ▲ 1
    Entropy Decoding in Optillm and Early Results on GSM8k
    story by codelion · 2024-10-20 · 0 comments
  18. ▲ 1
    Unbounded Context with Memory
    story by codelion · 2024-10-12 · 1 comment
  19. ▲ 2
    Optillm: An Optimizing Inference Proxy with Plugins
    story by codelion · 2024-10-05 · 1 comment
  20. ▲ 2
    Cot Decoding – Eliciting Reasoning from LLMs
    story by codelion · 2024-09-24 · 0 comments
  21. This is good I also had worked on something similar in optillm - https://github.com/codelion/optillm . You can do this with any LLM and several optimization techniques (including cot_reflection) like mcts, plansearch, moa etc.
    comment by codelion · 2024-09-16 · in g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains
  22. …spent some time on 2) and implemented several approaches in this open source optimising llm proxy - https://github.com/codelion/optillm In my experience it does work quite well, but we probably need different techniques for different tasks.
    comment by codelion · 2024-09-13 · in Notes on OpenAI's new o1 chain-of-thought models