ShelbyJenkins/llm_utils
View on GitHub ↗ ★ 47 Rust
llm_utils: Basic LLM tools, best practices, and minimal abstraction.
3 mentions on Hacker News by 1 person: 0 stories, 3 comments · first 2024-06-18 · latest 2025-04-13
20082026
-
…makes me feel better about spending so much time implementing this balanced text chunker last year. https://github.com/ShelbyJenkins/llm_utils It splits an input text into equal sized chunks using DFS and parallelization (rayon) to do so…
comment by J_Shelby_J · 2025-04-13 · in Show HN: Chonky – a neural approach for text semantic chunking -
…counting tokens. I guess it would make sense to add a function to convert tokens to Dollars. [1] https://github.com/ShelbyJenkins/llm_utils/tree/main/src/mod... [2] https://github.com/ShelbyJenkins/llm_utils/blob/main/src/pro...
comment by J_Shelby_J · 2024-06-18 · in Show HN: Token price calculator for 400+ LLMs -
…I also support non-open ai tokenizers. So adding a price calculator function would be trivial. [1] https://github.com/ShelbyJenkins/llm_utils
comment by J_Shelby_J · 2024-06-18 · in Show HN: Token price calculator for 400+ LLMs