GUI Automation Tools for Mac OS X

I've received a lot of inquiries recently from people who are interested in controlling Mac OS X's GUI. First, let me say that there are some limitations inherent to most GUI automation processes. Here are the top five that I see most often...

  1. GUI automation is interface-reliant. It expects the interface to always be the same. If it changes, the process must be updated.
  2. GUI automation can sometimes be error prone. For example, changing the window layering on screen, accidentally clicking the mouse, and so forth, can cause some GUI automation tools to become confused.
  3. Tracking down the cause of a GUI automation error or problem is not always straightforward. Locating "button 12" of "window 500", for example, isn't as easy as it sounds.
  4. GUI automation tools typically have a limited ability to perform branching, i.e. looking at something, and making a decision about how to proceed.
  5. While GUI automation tools may be able to click around, select menus, and type keystrokes, they aren't good for "processing" actual data. For example, don't expect GUI automation to retrieve content from a database, bring it into an InDesign document, construct a catalog, output it to PDF, and send it to your printer.

For simple tasks, GUI automation may work just fine. For complex tasks, it may not. In these situations, you may need something much more robust, such as AppleScript, or AppleScript+GUI automation, or some other tool.

So, strictly with regard to GUI automation, there are a number of tools that can help you get the job done…

Automator - Automator has the ability to record manual tasks, and play them back as part of a workflow. Here are some resources I've released in the past to get you started…

Third-Party Automator Actions - My Automator Extension Action Pack includes several GUI automation actions, including:

  • Click Safari Web Form Button
  • Click Safari Web Form Checkbox
  • Click Safari Web Form Radio Button
  • Set Safari Web Form Text or Popup Field
  • Type Keystroke

Learn more or download a demo of my Automator actions here.

QuicKeys - An automation tool that can simulate user interaction by clicking buttons, selecting menus, and more. Essentially, a macro utility. Learn more on the QuicKeys website and in the following episode of my podcast...

iKey - Another macro utility for Mac OS X, allowing you to automate keystrokes, mouse clicks, and more. Learn more on the iKey website and in the following episode of my podcast...

Sikuli - An interesting visual automation tool for Mac OS X, which works from screenshots. Learn more on the Project Sikuli website.

Fake - A browser for automating web tasks, such as filling out forms, testing websites, and more. Learn more on the website.

Eggplant - A visual, screen-shot-based automated testing tool, designed for automating processes across multiple machines (even PCs) through VPN. Learn more on the Eggplant website.

AppleScript Editor - Used to write AppleScripts, but also includes the ability to record manual tasks in certain applications as a script for later playback. Unfortunately, recordable applications are few and far between, but some include BBEdit, Fetch, and the Mac OS X Finder. Learn more in the following episode of my podcast...

If you are into AppleScript, check out my MacTech column on User Interface Scripting.

MacTech AppleScript Essentials Column > AppleScript in Snow Leopard

October, 2009 - AppleScript in Snow Leopard.

With the release of Snow Leopard, AppleScript developers will no doubt be in store for a few surprises. While the AppleScript language itself hasn't changed too much, a variety of other enhancements are welcome, but will take some getting used to.

[Read more at]

Mac Automation Made Simple Video Podcast > Introducing AppleScript and Script Editor

Episode 17
Date: 03.03.2009
Duration: 00:09:06
Introducing AppleScript and Script Editor
In this episode of Mac Automation Made Simple, Ben Waldie provides an introduction to the primary AppleScript components in Mac OS X, including the AppleScript Utility, Folder Actions Setup application, example scripts, and Script Editor..
[View in Tunes...] [View at]

X-Ray Magazine Scripting Quark Article > Getting to Know QuarkXPress’ AppleScript Terminology

Vol. 3, Num. 2 - Getting to Know QuarkXPress' AppleScript Terminology.

For both beginning and experienced scripters, implementing AppleScript automation intoa workflow frequently brings with it a learning curve. This is because every scriptable application extends the base AppleScript language with its own set of unique terminology. If you are attempting to script an application that you have not automated before, then you will first need to familiarize yourself with that application’s AppleScript terminology. As you begin to script the application, you will also become more comfortable in doing so. Before long, scripting the application will become second nature. Automating QuarkXPress with AppleScript is no different. [Read more at X-Ray Magazine online...]

MacTech AppleScript Essentials Column > The "NEW" Script Editor

July 2004 - The "NEW" Script Editor.

A few months ago, Dave Mark wrote a couple of articles about AppleScript. These articles touched briefly on topics including setting up the script menu in Mac OS X Panther, performing some basic Finder scripting, and accessing application dictionaries. This aims to be the first in a series of articles geared toward providing a more in-depth look at AppleScript. [Read more at]