How I gave AI full control of a Windows machine using LibreAutomate

[birdywen]

Hi everyone,
I want to share a project that started as a simple experiment and turned into something genuinely powerful. And I want to thank Gintaras — without LibreAutomate and his 20+ years of dedication, none of this would have been possible.
The goal: Give an AI (Claude) the ability to see, control, and automate a Windows desktop — not through pre-built plugins, but with true, unlimited capability.
The setup:
I have three machines working together:

• A Mac running Claude as the “brain”
  
• An Oracle Linux cloud server
  
• A Windows Surface Pad running LibreAutomate as the “eyes and hands”
  

The Surface Pad sits always-on with a bore tunnel (fixed port) exposing LibreAutomate’s HTTP Server to the outside. Claude connects from the Mac through this tunnel.
How it works:
We started with the HTTP Server recipe from the LA cookbook — the one that maps URL paths to C# static methods. We added basic endpoints:

Code:

Copy      Help

Ping

,

Code:

Copy      Help

ScreenInfo

,

Code:

Copy      Help

ListWindows

,

Code:

Copy      Help

ClickAt

,

Code:

Copy      Help

TypeText

,

Code:

Copy      Help

SendKeys

,

Code:

Copy      Help

RunProgram

.
But the real breakthrough came when we added a Code endpoint, using Gintaras’s

Code:

Copy      Help

CsScript

class from this forum post. This uses Roslyn to compile and execute arbitrary C# code at runtime.
Now Claude can POST any C# code to the Surface and it gets compiled and executed instantly. The code has full access to the entire Au.dll API — all 280+ classes. That means:

• Code:

  Copy      Help

  wnd
  — find, move, resize, close any window
  
• Code:

  Copy      Help

  elm
  — read and interact with any UI element in any application
  
• Code:

  Copy      Help

  keys
  /
  
  Code:

  Copy      Help

  mouse
  — type text, send hotkeys, click anywhere
  
• Code:

  Copy      Help

  ocr
  — read text from the screen
  
• Code:

  Copy      Help

  uiimage
  — find images on screen
  
• Code:

  Copy      Help

  clipboard
  — read/write clipboard
  
• Code:

  Copy      Help

  filesystem
  — full file system access
  
• Code:

  Copy      Help

  process
  /
  
  Code:

  Copy      Help

  run
  — launch and manage programs
  
• Code:

  Copy      Help

  screen
  /
  
  Code:

  Copy      Help

  CaptureScreen
  — screen information and screenshots
  
• And hundreds more…
  

A concrete example:
Claude sends this HTTP request:

Code:

Copy      Help

POST http://bore.pub:14455/Code
Content-Type: text/plain

var asm = System.Reflection.Assembly.LoadFrom(@"C:\Program Files\LibreAutomate\Au.dll");
var types = asm.GetExportedTypes();
return $"Au.dll has {types.Length} public types";

And gets back:

Code:

Copy      Help

{"success":true,"result":"Au.dll has 280 public types"}

It can query the screen:

Code:

Copy      Help

var w = wnd.active;
return w.Name + " | " + w.Rect.ToString();

It can check disk space, list windows, click buttons, type text — anything. The AI writes C# on the fly, LibreAutomate compiles and runs it in ~100ms, and returns the result. No pre-defined tool list. No plugins. No boundaries.
Why LibreAutomate is uniquely suited for this:
I evaluated many automation tools. Nothing else comes close. Here’s why:

1. Depth of API — Au.dll covers four layers: Win32 API, COM, UI Automation, and keyboard/mouse simulation. If one layer can’t reach something, another can.
   
2. Full .NET ecosystem — It’s real C#, not a toy scripting language. NuGet packages, LINQ, async/await, everything works.
   
3. The HTTP Server recipe — This was the key enabler. A built-in way to expose C# functions over HTTP, with clean request/response handling.
   
4. CsScript / Roslyn integration — Runtime compilation turns LibreAutomate from a “toolbox” into a “universal execution engine.”
   
5. 20 years of battle-testing — From Quick Macros to LibreAutomate, Gintaras has solved every edge case. The API is powerful yet concise. It just works.
   

The architecture comparison:
Many people use MCP (Model Context Protocol) to connect AI to tools. MCP defines fixed tools — you get 10 tools, you can do 10 things. Our approach is fundamentally different. We give the AI one endpoint (Code) and the entire .NET + Au.dll universe behind it. It’s like giving someone a full kitchen instead of a cookbook with 10 recipes.
What’s next:

• Screenshot endpoint — so the AI can “see” the screen
  
• File transfer — pull files from Windows back to Mac
  
• Long-running task support — async execution with polling
  

Thank you, Gintaras
I want to end with genuine gratitude. Gintaras has been building automation tools since 2002. He responds to forum posts personally. He shares code like CsScript freely. LibreAutomate is free and open source. The quality of Au.dll is extraordinary — 280+ classes, each one polished and well-designed.
Without LibreAutomate, I would have had to cobble together AutoHotkey, PyAutoGUI, Windows API wrappers, and a dozen other tools — and the result would have been fragile and limited. With LibreAutomate, I got everything in one coherent package, with an HTTP server built in, and Roslyn compilation ready to go.
This project proves that LibreAutomate isn’t just a macro recorder or a simple automation tool. It’s a full operating system-level automation framework that can serve as the hands and eyes for an AI brain. That’s an incredible achievement for a tool made by one person over two decades.
Thank you, Gintaras. You made this possible.