GUI Automation Tools for Mac OS X

I've received a lot of inquiries recently from people who are interested in controlling Mac OS X's GUI. First, let me say that there are some limitations inherent to most GUI automation processes. Here are the top five that I see most often...

  1. GUI automation is interface-reliant. It expects the interface to always be the same. If it changes, the process must be updated.
  2. GUI automation can sometimes be error prone. For example, changing the window layering on screen, accidentally clicking the mouse, and so forth, can cause some GUI automation tools to become confused.
  3. Tracking down the cause of a GUI automation error or problem is not always straightforward. Locating "button 12" of "window 500", for example, isn't as easy as it sounds.
  4. GUI automation tools typically have a limited ability to perform branching, i.e. looking at something, and making a decision about how to proceed.
  5. While GUI automation tools may be able to click around, select menus, and type keystrokes, they aren't good for "processing" actual data. For example, don't expect GUI automation to retrieve content from a database, bring it into an InDesign document, construct a catalog, output it to PDF, and send it to your printer.

For simple tasks, GUI automation may work just fine. For complex tasks, it may not. In these situations, you may need something much more robust, such as AppleScript, or AppleScript+GUI automation, or some other tool.

So, strictly with regard to GUI automation, there are a number of tools that can help you get the job done…

Automator - Automator has the ability to record manual tasks, and play them back as part of a workflow. Here are some resources I've released in the past to get you started…

Third-Party Automator Actions - My Automator Extension Action Pack includes several GUI automation actions, including:

  • Click Safari Web Form Button
  • Click Safari Web Form Checkbox
  • Click Safari Web Form Radio Button
  • Set Safari Web Form Text or Popup Field
  • Type Keystroke

Learn more or download a demo of my Automator actions here.

QuicKeys - An automation tool that can simulate user interaction by clicking buttons, selecting menus, and more. Essentially, a macro utility. Learn more on the QuicKeys website and in the following episode of my podcast...

iKey - Another macro utility for Mac OS X, allowing you to automate keystrokes, mouse clicks, and more. Learn more on the iKey website and in the following episode of my podcast...

Sikuli - An interesting visual automation tool for Mac OS X, which works from screenshots. Learn more on the Project Sikuli website.

Fake - A browser for automating web tasks, such as filling out forms, testing websites, and more. Learn more on the website.

Eggplant - A visual, screen-shot-based automated testing tool, designed for automating processes across multiple machines (even PCs) through VPN. Learn more on the Eggplant website.

AppleScript Editor - Used to write AppleScripts, but also includes the ability to record manual tasks in certain applications as a script for later playback. Unfortunately, recordable applications are few and far between, but some include BBEdit, Fetch, and the Mac OS X Finder. Learn more in the following episode of my podcast...

If you are into AppleScript, check out my MacTech column on User Interface Scripting.

MacTech AppleScript Essentials Column > Providing Progress Feedback During Script Execution

January, 2006 - Providing Progress Feedback During Script Execution.

Many AppleScripts do not provide progress updates to the user during processing. Most of the time, when a script is run, it simply performs the appropriate tasks "behind the scenes," so to speak. If run as an application, a script may appear in the Dock when launched. However, this hardly provides detailed information to the user about what is actually occurring. Sometimes, a script may not need to provide progress updates to the user. However, there are situations when providing such feedback is a good idea.

In this month's column, we will walk through the process of creating a script that will provide visual progress information to the user during processing. The script we will create will save selected email messages in Mail as text files into a user-specified output folder. Since the script will have the ability to process multiple selected email messages, we will write our code to provide a visual indication to the user of which message is currently being processed. Once you learn how to provide this type of feedback, then you can begin integrating this same technique into your other scripts, making them more user friendly. [Read more at]

MacTech AppleScript Essentials Column > Building Table Views with AppleScript Studio

December, 2005 - Building Table Views with AppleScript Studio.

Back in my April column, I provided an introduction to AppleScript Studio, a technology that is a part of Xcode and Interface Builder, Mac OS X's powerful integrated development environment. Using AppleScript Studio, AppleScript developers are able to create fully native Mac OS X applications, complete with user interfaces. By constructing scripts with interfaces, developers can build complex and powerful user friendly AppleScript solutions that have the same look and feel of any other Mac OS X application.

In addition to building stand-alone AppleScript solutions, AppleScript Studio can also now be used to create Automator actions, for use with Mac OS X 10.4's popular new Automator application. In my August column, I walked through the process of building of a simple AppleScript Studio-based Automator action.

This month, we will create a new AppleScript Studio-based project that is a little more complex, and incorporates a table view into its interface. This application, which I call Enable|Disable Mail Accounts, will display a list of email accounts in Mail to the user, and allow the user to toggle whether the accounts are enabled or disabled. [Read more at]

MacTech AppleScript Essentials Column > User Interface Scripting

June, 2005 - User Interface Scripting.

As we have seen in the past, AppleScript is a great tool for creating some pretty amazing automated workflows. When implementing an AppleScript-based workflow, you are really only limited by your imagination, and by the AppleScript support that is available in the applications that you want to automate.
Many scriptable applications offer enough AppleScript support for the types of tasks that you would want to automate the most. However, at times, you may find yourself needing to script an application with limited AppleScript support, and that one task you really need to automate just is not accessible through scripting. Or, worse yet, the application you want to automate is not scriptable at all!

How do you handle these limitations? Do you simply give up? I think not. For starters, you might check around to see if there are other scriptable applications that can be substituted to automate the same task.

Another option is to consider trying to automate the application by writing AppleScript code that interacts directly with the application's interface itself. Fortunately, with the release of Mac OS X 10.3, Apple introduced a new AppleScript feature, user interface scripting, or UI Scripting, which can be used to do just that. [Read more at]

MacTech AppleScript Essentials Column > Introduction to AppleScript Studio

April, 2005 - Introduction to AppleScript Studio.

In past articles, we have discussed many AppleScript-related topics. However, one of the topics that we have yet to touch on is AppleScript Studio, which is quickly becoming the tool of choice for AppleScript developers. [Read more at]

MacTech AppleScript Essentials Column > User Interaction Basics

August 2004 - User Interaction Basics.

Last month, we looked at some of the features of the new Script Editor, which was released with Mac OS X Panther (10.3). Now we are going to get started with actually writing some AppleScript code! This month's article will explain how, with only minimal code, you can update your AppleScripts to interact with the user. We will primarily focus on displaying dialogs and prompting for data. [Read more at]