Hacker GitHub

← All repos

OpenAdaptAI/OpenAdapt

View on GitHub ↗ ★ 1,860 Python

For EHRs, payer portals, and any app your team clicks through. A person or an agent shows the task; OpenAdapt turns it into a local program that repeats the steps with no model calls, checks each save, and stops when something doesn't match.

59 mentions on Hacker News by 8 people: 6 stories, 53 comments · first 2023-07-03 · latest 2025-12-04

Mentions per year 59 mentions · busiest year 2024 (39)
20082026
AllStoriesComments NewestTopOldest
  1. …could watch my daily habits and offer automation for recurring things. We're working on it at https://github.com/openadaptai/openadapt .
    comment by abrichr · 2025-12-04 · in Microsoft drops AI sales targets in half after salespeople miss their quotas
  2. You might be interested in https://github.com/OpenAdaptAI/OpenAdapt
    comment by abrichr · 2025-05-13 · in FastVLM: Efficient vision encoding for vision language models
  3. Thanks for sharing! There is an open issue in OpenAdapt to implement COM support: https://github.com/OpenAdaptAI/OpenAdapt/issues/873 This could be a valuable reference.
    comment by abrichr · 2025-03-03 · in Show HN: Open-Source Windows AI assistant that uses Word, Excel through COM
  4. This is exactly why in https://github.com/OpenAdaptAI/OpenAdapt : 1. Our goal is to automate repetitive tasks (not simple one off tasks that need human input, but repetitive tasks that human shouldn't be doing in the first place), and 2.…
    comment by abrichr · 2025-01-18 · in Skyvern Browser Agent 2.0: How We Reached State of the Art in Evals
  5. …a process for which it doesn't know the steps or the rules This is exactly the motivation behind https://github.com/OpenAdaptAI/OpenAdapt : so that users can demonstrate their desktop workflows to AI models step by step (without worrying…
    comment by abrichr · 2025-01-12 · in AI founders will learn the bitter lesson
  6. We designed https://github.com/OpenAdaptAI/OpenAdapt to ground the agent to human demonstrations, so that if you as a human don't engage in the dark pattern, neither will the agent.
    comment by abrichr · 2025-01-05 · in AI's next leap requires intimate access to your digital life
  7. The reason I created https://github.com/OpenAdaptAI/OpenAdapt is because I had the same concern, and wanted to create a way for users and organizations to avoid exfiltrating their data while still participating in the promise of AI. I…
    comment by abrichr · 2025-01-05 · in AI's next leap requires intimate access to your digital life
  8. See https://github.com/OpenAdaptAI/OpenAdapt for an open source alternative that runs segmentation locally.
    comment by abrichr · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
  9. See https://github.com/OpenAdaptAI/OpenAdapt for an open source alternative that includes a desktop app.
    comment by abrichr · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
  10. …to be concerned about privacy implications of sending screenshots of the desktop That's why in https://github.com/OpenAdaptAI/OpenAdapt we've built in several state-of-the-art PII/PHI scrubbers.
    comment by abrichr · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
  11. We use operating system accessibility APIs when available in https://github.com/OpenAdaptAI/OpenAdapt .
    comment by abrichr · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
  12. See https://github.com/OpenAdaptAI/OpenAdapt for an open source implementation that includes a desktop app OOTB.
    comment by abrichr · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
  13. See https://github.com/OpenAdaptAI/OpenAdapt for an open source alternative that includes operating system accessibility API data and DOM information (along with bitmaps) where available. We are also planning on extracting runtime…
    comment by abrichr · 2024-10-22 · in Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku
  14. > Doesn’t work well for screen based content in general. It's not perfect, but it works: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 > the most recent latent space pod Link: https://www.latent.space/p/sam2
    comment by abrichr · 2024-08-12 · in Segment Anything Model and Friends
  15. We are using Segment Anything Model at OpenAdapt for exactly this purpose: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 It works surprisingly well despite the fact that the model was not trained on this type of data.
    comment by abrichr · 2024-08-12 · in Segment Anything Model and Friends
  16. We use SAM to segment GUI elements in https://github.com/OpenAdaptAI/OpenAdapt
    comment by abrichr · 2024-07-30 · in SAM 2: Segment Anything in Images and Videos
  17. Related: https://github.com/OpenAdaptAI/OpenAdapt
    comment by abrichr · 2024-06-26 · in Show HN: a Rust lib to trigger actions based on your screen activity (with LLMs)
  18. …you for the link! Our initial testing suggests MiniCPM outperforms InternVL for GUI understanding: https://github.com/OpenAdaptAI/OpenAdapt/issues/637#issuecom... (InternVL appears to hallucinate more.)
    comment by abrichr · 2024-05-29 · in Llama 3-V: Matching GPT4-V with a 100x smaller model and 500 dollars
  19. It's very reliable for GUI segment understanding; see e.g. https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (scroll down to `gpt-4-vision-preview`).
    comment by abrichr · 2024-05-29 · in Llama 3-V: Matching GPT4-V with a 100x smaller model and 500 dollars
  20. We have Linux support on the roadmap in https://github.com/OpenAdaptAI/OpenAdapt . OpenAdapt has similar functionality, except: - it's open source - it only records when you explicitly tell it to - it has multiple PII/PHI scrubbing…
    comment by abrichr · 2024-05-22 · in Why MSFT Copilot+ and AI PCs are the final nail in the coffin of open computing
  21. …records when you explicitly tell it to - it has multiple PII/PHI scrubbing providers built in (see https://github.com/OpenAdaptAI/OpenAdapt?tab=readme-ov-file#... ) - the reason for recording is to automate tasks in desktop apps - it's…
    comment by abrichr · 2024-05-22 · in New Windows AI feature records everything you've done on your PC
  22. Excited to test how this performs compared to MiniCPMv2, especially when analyzing GUI images: https://github.com/OpenAdaptAI/OpenAdapt/issues/637
    comment by abrichr · 2024-05-15 · in PaliGemma: Open-Source Multimodal Model by Google
  23. Congratulations on shipping! In https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt... we use FastSAM to first segment the UI elements, then have the LLM describe each segment individually. This seems to work quite well; see…
    comment by abrichr · 2024-05-15 · in Show HN: Tarsier – Vision utilities for web interaction agents
  24. > where are my agents? https://github.com/OpenAdaptAI/OpenAdapt/
    comment by abrichr · 2024-05-13 · in GPT-4o
  25. …anyone is interested in running "teach mode" directly on their desktop, we're building this over at https://github.com/OpenAdaptAI/OpenAdapt .
    comment by abrichr · 2024-05-05 · in Rabbit R1 can be run on a Android device
  26. > teach mode We're building this over at https://github.com/OpenAdaptAI/OpenAdapt . OpenAdapt learns to automate tasks in desktop apps by observing human demonstrations. Early demo: https://twitter.com/abrichr/status/1784307190062342237…
    comment by abrichr · 2024-05-01 · in Rabbit R1: Barely Reviewable [video]
  27. …has said the device will be able to learn any app, if you teach it. We're building this over at https://github.com/OpenAdaptAI/OpenAdapt . OpenAdapt learns to automate tasks in desktop apps by observing human demonstrations. Early demo:…
    comment by abrichr · 2024-04-30 · in I Witnessed the Future of AI, and It's a Broken Toy
  28. Thanks! OpenAdapt already supports audio recording during demonstration ( https://github.com/OpenAdaptAI/OpenAdapt/pull/346 ). Perhaps I misunderstood — can you please clarify your suggestion?
    comment by abrichr · 2024-04-29 · in Memary: Open-Source Longterm Memory for Autonomous Agents
  29. Very interesting, thank you for making this available! At OpenAdapt ( https://github.com/OpenAdaptAI/OpenAdapt ) we are looking into using pm4py ( https://github.com/pm4py ) to extract a process graph from a recording of user actions. I…
    comment by abrichr · 2024-04-29 · in Memary: Open-Source Longterm Memory for Autonomous Agents
  30. Thank you for making this available! Check out https://github.com/OpenAdaptAI/OpenAdapt for a cross platform (Mac and Windows) open source library that learns to perform tasks in desktop apps by observing human demonstrations. We believe a…
    comment by abrichr · 2024-04-29 · in OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computers
  31. https://github.com/OpenAdaptAI/OpenAdapt runs locally on your machine to automate repetitive tasks in desktop apps. We have plans to build a mobile interface as well.
    comment by abrichr · 2024-04-24 · in A morning with the Rabbit R1: a fun, funky, unfinished AI gadget
  32. See https://github.com/OpenAdaptAI/OpenAdapt for an alternative that works with desktop GUIs.
    comment by abrichr · 2024-04-24 · in Rabbit R1 source code [part 1]
  33. For an open source alternative, check out https://github.com/OpenAdaptAI/OpenAdapt . We combine Segment Anything Model with GPT4-V to understand recordings of workflows in desktop user interfaces, then replay them according to the user's…
    comment by abrichr · 2024-04-24 · in Rabbit R1 source code [part 1]
  34. Not mentioned: learning from demonstration. This is the approach we are taking at https://github.com/OpenAdaptAI/OpenAdapt .
    comment by abrichr · 2024-04-22 · in Survey Study on AI Agent Architectures (2024)
  35. …`gpt-4-1106-vision-preview` and simply prompting the model to return json, with excellent results: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress).
    comment by abrichr · 2024-04-09 · in GPT-4 Turbo with Vision Generally Available
  36. …we're in Canada, but GPT-4-V works extremely well (when combined with Segment Anything). See: https://github.com/OpenAdaptAI/OpenAdapt/pull/610 (work in progress). Unfortunately we can't compare it to ScreenAI directly since as far as I…
    comment by abrichr · 2024-04-09 · in ScreenAI: A visual LLM for UI and visually-situated language understanding
  37. …combining Segment Anything Model (SAM) with GPT-4 for screen understanding. Work-in-progress: https://github.com/OpenAdaptAI/OpenAdapt/pull/610
    comment by abrichr · 2024-04-09 · in ScreenAI: A visual LLM for UI and visually-situated language understanding
  38. https://github.com/OpenAdaptAI/OpenAdapt is designed for all desktop applications, including browsers. We're also working on a chrome extension to support reading/writing directly to DOM: https://github.com/OpenAdaptAI/OpenAdapt/pull/364
    comment by abrichr · 2024-03-15 · in LaVague: Open-source Large Action Model to automate Selenium browsing
  39. Congratulations on shipping! Check out https://github.com/OpenAdaptAI/OpenAdapt for an open source (MIT license) alternative that also works on desktop (including Citrix!)
    comment by abrichr · 2024-03-15 · in Show HN: Skyvern – Browser automation using LLMs and computer vision
  40. Or let https://github.com/OpenAdaptAI/OpenAdapt do the boring tasks for you!
    comment by abrichr · 2024-03-13 · in A generalist AI agent for 3D video games
  41. …chatbots, here's your chance! We're using them to automate repetitive tasks in GUI workflows. See https://github.com/OpenAdaptAI/OpenAdapt for more. Python / TypeScript (but open to other languages).
    comment by abrichr · 2024-02-01 · in Ask HN: Looking for a project to volunteer on? (February 2024)
  42. Thank you for your perspective! (We are looking into both for https://github.com/OpenAdaptAI/OpenAdapt )
    comment by abrichr · 2023-12-15 · in Prompt engineering
  43. This is exactly why in https://github.com/OpenAdaptAI/OpenAdapt we have implemented three separate PII scrubbing providers. Congrats to the op on shipping!
    comment by abrichr · 2023-12-12 · in Show HN: Open-source macOS AI copilot using vision and voice
  44. Thanks for the tip! Automating repetitive GUI workflows is the goal of https://github.com/OpenAdaptAI/OpenAdapt
    comment by abrichr · 2023-11-10 · in Using GPT-4 Vision with Vimium to browse the web
  45. Agreed! This is part of the motivation behind https://github.com/OpenAdaptAI/OpenAdapt
    comment by abrichr · 2023-11-10 · in Using GPT-4 Vision with Vimium to browse the web
  46. This type of use case is exactly why are building https://github.com/OpenAdaptAI/OpenAdapt
    comment by abrichr · 2023-11-10 · in Using GPT-4 Vision with Vimium to browse the web
  47. This is exactly why in https://github.com/OpenAdaptAI/OpenAdapt we have put a lot of emphasis on scrubbing screenshots before sharing them with the model, and built tools to visualize the results.
    comment by abrichr · 2023-11-09 · in Sharing Screen with GPT 4 vision
  48. Done: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/LICENSE
    comment by abrichr · 2023-10-19 · in Fuyu-8B: A multimodal architecture for AI agents
  49. "fully open source", but there is no license? https://github.com/OpenAdaptAI/OpenAdapt/blob/30581e47fa9aec... https://github.com/OpenAdaptAI/OpenAdapt/issues/246 And Fuyu is under a non-commercial license, so there's not much to be done…
    comment by coder543 · 2023-10-19 · in Fuyu-8B: A multimodal architecture for AI agents
  50. …available! For anyone interested in contributing to a fully open source alternative, join us at https://github.com/OpenAdaptAI/OpenAdapt Lots of interesting work to be done, including integrating with Fuyu-8B!
    comment by abrichr · 2023-10-19 · in Fuyu-8B: A multimodal architecture for AI agents