Computer Tools¶
The ComputerTool provides system-level automation capabilities for mouse and keyboard control, enabling LLMs to interact with applications and perform desktop automation tasks.
Overview¶
ComputerTool allows LLMs to:
- Control mouse movements and clicks
- Type text and keyboard shortcuts
- Scroll windows
- Automate desktop workflows
- Fill forms and interact with UI elements
Platform Compatibility
Computer automation features work best on macOS with accessibility permissions enabled. Other platforms may require additional configuration.
Actions¶
Mouse Actions¶
Mouse Click¶
Click at specific coordinates with various button options.
computer = SharedTools::Tools::ComputerTool.new
# Left click
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_CLICK,
coordinate: { x: 100, y: 200 },
mouse_button: "left"
)
# Right click
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_CLICK,
coordinate: { x: 100, y: 200 },
mouse_button: "right"
)
# Double click
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_DOUBLE_CLICK,
coordinate: { x: 100, y: 200 },
mouse_button: "left"
)
# Triple click
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_TRIPLE_CLICK,
coordinate: { x: 100, y: 200 }
)
Parameters:
- coordinate: Hash with x and y keys (required)
- mouse_button: "left" (default), "right", or "middle"
Use the MOUSE_DOUBLE_CLICK and MOUSE_TRIPLE_CLICK actions for multiple clicks.
Mouse Move¶
Move mouse to specific coordinates.
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_MOVE,
coordinate: { x: 500, y: 300 }
)
Mouse Position¶
Get current mouse position.
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_POSITION
)
puts "Mouse is at: #{result[:x]}, #{result[:y]}"
Keyboard Actions¶
Type Text¶
Type text at current cursor position.
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::TYPE,
text: "Hello, World!"
)
Press Key¶
Press keyboard keys including modifiers.
# Single key
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "Return"
)
# Keyboard shortcut
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "cmd+c" # Copy on macOS
)
# Multiple modifiers
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "cmd+shift+v" # Paste and match style
)
Common Keys:
- Return, Enter, Tab, Escape
- Backspace, Delete, Space
- ArrowUp, ArrowDown, ArrowLeft, ArrowRight
- cmd (macOS), ctrl (Windows/Linux)
- shift, alt, option
Hold Key¶
Hold a key for a specific duration.
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::HOLD_KEY,
text: "shift",
duration: 2 # Hold for 2 seconds
)
Scroll Action¶
Scroll in a window.
# Scroll down
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::SCROLL,
scroll_direction: "down",
scroll_amount: 5
)
# Scroll up
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::SCROLL,
scroll_direction: "up",
scroll_amount: 3
)
Wait Action¶
Wait for a specified duration.
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::WAIT,
duration: 2 # Wait 2 seconds
)
Complete Examples¶
Form Automation¶
computer = SharedTools::Tools::ComputerTool.new
# Click on name field
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_CLICK,
coordinate: { x: 300, y: 200 }
)
# Type name
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::TYPE,
text: "John Doe"
)
# Tab to next field
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "Tab"
)
# Type email
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::TYPE,
text: "john@example.com"
)
# Submit form
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "Return"
)
Text Selection and Copy¶
# Triple-click to select line
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_TRIPLE_CLICK,
coordinate: { x: 400, y: 300 }
)
# Copy
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "cmd+c"
)
# Wait a moment
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::WAIT,
duration: 0.5
)
# Click elsewhere
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_CLICK,
coordinate: { x: 400, y: 500 }
)
# Paste
computer.execute(
action: SharedTools::Tools::ComputerTool::Action::KEY,
text: "cmd+v"
)
Working with Drivers¶
Default Driver¶
With no driver:, ComputerTool uses SharedTools::Tools::Computer::MacDriver on macOS when the macos gem is installed. Anywhere else it raises LoadError unless you pass a driver:
Custom Drivers¶
Subclass SharedTools::Tools::Computer::BaseDriver and implement its methods. ComputerTool passes the mouse button to the driver as button:, and calls type(text:) for typing:
# macOS example using cliclick
class CliclickDriver < SharedTools::Tools::Computer::BaseDriver
def mouse_click(coordinate:, button:)
system("cliclick", "c:#{coordinate[:x]},#{coordinate[:y]}")
end
def type(text:)
system("cliclick", "t:#{text}")
end
# ... key, hold_key, mouse_position, mouse_move, mouse_down, mouse_up,
# scroll, screenshot (see Computer::BaseDriver)
end
computer = SharedTools::Tools::ComputerTool.new(
driver: CliclickDriver.new(display_width: 1920, display_height: 1080, display_number: 1)
)
BaseDriver implements mouse_double_click and mouse_triple_click by calling mouse_click two or three times.
Best Practices¶
Coordinate Finding¶
- Use Screenshot Tools: Take screenshots and identify coordinates
- Add Delays: System UIs may need time to respond
- Verify UI State: Check if elements are ready before clicking
Reliability¶
# Add waits between actions
# Actions are strings ("mouse_click"), not symbols
computer.execute(action: "mouse_click", coordinate: {x: 100, y: 200})
computer.execute(action: "wait", duration: 0.5)
computer.execute(action: "type", text: "Hello")
Error Handling¶
result = computer.execute(
action: SharedTools::Tools::ComputerTool::Action::MOUSE_CLICK,
coordinate: { x: 100, y: 200 }
)
if result[:error]
puts "Click failed: #{result[:error]}"
else
puts "Click succeeded"
end
Troubleshooting¶
Accessibility Permissions¶
On macOS, enable accessibility permissions:
- System Preferences → Security & Privacy → Privacy
- Select "Accessibility"
- Add your application/terminal
Coordinate Issues¶
- Screen coordinates are from top-left (0,0)
- Use screenshot tools to find exact coordinates
- Consider screen resolution and scaling
Timing Problems¶
- Add waits between actions
- Increase duration for slower UIs
- Check if dialogs are modal
See Also¶
- BrowserTool - For web automation
- Examples - More examples
- Driver Guide - Custom driver creation