GUI Automation Tools for Mac OS X

I've received a lot of inquiries recently from people who are interested in controlling Mac OS X's GUI. First, let me say that there are some limitations inherent to most GUI automation processes. Here are the top five that I see most often...

  1. GUI automation is interface-reliant. It expects the interface to always be the same. If it changes, the process must be updated.
  2. GUI automation can sometimes be error prone. For example, changing the window layering on screen, accidentally clicking the mouse, and so forth, can cause some GUI automation tools to become confused.
  3. Tracking down the cause of a GUI automation error or problem is not always straightforward. Locating "button 12" of "window 500", for example, isn't as easy as it sounds.
  4. GUI automation tools typically have a limited ability to perform branching, i.e. looking at something, and making a decision about how to proceed.
  5. While GUI automation tools may be able to click around, select menus, and type keystrokes, they aren't good for "processing" actual data. For example, don't expect GUI automation to retrieve content from a database, bring it into an InDesign document, construct a catalog, output it to PDF, and send it to your printer.

For simple tasks, GUI automation may work just fine. For complex tasks, it may not. In these situations, you may need something much more robust, such as AppleScript, or AppleScript+GUI automation, or some other tool.

So, strictly with regard to GUI automation, there are a number of tools that can help you get the job done…

Automator - Automator has the ability to record manual tasks, and play them back as part of a workflow. Here are some resources I've released in the past to get you started…

Third-Party Automator Actions - My Automator Extension Action Pack includes several GUI automation actions, including:

  • Click Safari Web Form Button
  • Click Safari Web Form Checkbox
  • Click Safari Web Form Radio Button
  • Set Safari Web Form Text or Popup Field
  • Type Keystroke

Learn more or download a demo of my Automator actions here.

QuicKeys - An automation tool that can simulate user interaction by clicking buttons, selecting menus, and more. Essentially, a macro utility. Learn more on the QuicKeys website and in the following episode of my podcast...

iKey - Another macro utility for Mac OS X, allowing you to automate keystrokes, mouse clicks, and more. Learn more on the iKey website and in the following episode of my podcast...

Sikuli - An interesting visual automation tool for Mac OS X, which works from screenshots. Learn more on the Project Sikuli website.

Fake - A browser for automating web tasks, such as filling out forms, testing websites, and more. Learn more on the website.

Eggplant - A visual, screen-shot-based automated testing tool, designed for automating processes across multiple machines (even PCs) through VPN. Learn more on the Eggplant website.

AppleScript Editor - Used to write AppleScripts, but also includes the ability to record manual tasks in certain applications as a script for later playback. Unfortunately, recordable applications are few and far between, but some include BBEdit, Fetch, and the Mac OS X Finder. Learn more in the following episode of my podcast...

If you are into AppleScript, check out my MacTech column on User Interface Scripting.

MacTech AppleScript Essentials Column > User Interface Scripting

June, 2005 - User Interface Scripting.

As we have seen in the past, AppleScript is a great tool for creating some pretty amazing automated workflows. When implementing an AppleScript-based workflow, you are really only limited by your imagination, and by the AppleScript support that is available in the applications that you want to automate.
Many scriptable applications offer enough AppleScript support for the types of tasks that you would want to automate the most. However, at times, you may find yourself needing to script an application with limited AppleScript support, and that one task you really need to automate just is not accessible through scripting. Or, worse yet, the application you want to automate is not scriptable at all!

How do you handle these limitations? Do you simply give up? I think not. For starters, you might check around to see if there are other scriptable applications that can be substituted to automate the same task.

Another option is to consider trying to automate the application by writing AppleScript code that interacts directly with the application's interface itself. Fortunately, with the release of Mac OS X 10.3, Apple introduced a new AppleScript feature, user interface scripting, or UI Scripting, which can be used to do just that. [Read more at]

MacTech AppleScript Essentials Column > User Interaction Basics

August 2004 - User Interaction Basics.

Last month, we looked at some of the features of the new Script Editor, which was released with Mac OS X Panther (10.3). Now we are going to get started with actually writing some AppleScript code! This month's article will explain how, with only minimal code, you can update your AppleScripts to interact with the user. We will primarily focus on displaying dialogs and prompting for data. [Read more at]