Getting Sikuli Script working with Ruby(JRuby) on the Mac

Sikuli Script is one of only a handful of application that you can use to automate environments such as games, flash websites, or anything that takes a graphical approach to things. For me, getting it to support a Ruby application was key to doing what I wanted to do with it: the automation of a complex GUI.

Sikuli has a rich API and scripting language, and appears easy to use in Python, but is more complex to code to using Ruby, since it requires either JRuby or a Ruby/Java bridge and some trial and error. Since I’ve been through some of the pain in getting it working with Ruby, I’ve tried to shortcut the error for you by documenting some of my findings here.

This post looks at one approach to getting it up and running under Ruby. I tried a number of different tacks from a number of websites and found none of them to work properly. Either they were using a different version of Sikuli, or some outdated Ruby commands. This one works for me with JRuby. If you have success at getting it to work with standard Ruby using a Java bridge, please drop by and let me know.

This approach should also work for other OS’s, since the key thing is to set the CLASSPATH to the location of the jar files that Sikuli IDE installs. For the purposes of this example, I’m using the standard installation location on the Mac.

1. Grab the latest Sikuli IDE and install it from here.
2. Create a ruby file called sikuli.rb and put the following code into it

require 'rubygems'
require 'java'
$CLASSPATH << "/Applications/Sikuli-IDE.app/Contents/Resources/Java/"
#This is key to making it work under ruby. The path above is the default
#installation location on the Mac which contains all of the
#relevant sikuli jar files.
require "sikuli-script.jar"
java_import 'org.sikuli.script.Region'
java_import 'org.sikuli.script.Screen'
java_import 'org.sikuli.script.Settings'
java_import 'org.sikuli.script.SikuliEvent'
java_import 'org.sikuli.script.SikuliScript'
include Java
Sikuli=Java::OrgSikuliScript

3. Experiment and enjoy.

There is quite a bit of documentation for Python at sikuli.org, but as we are calling functions from Ruby, you may find yourself trying a few things out in order to get them to work as you expect. For example, I’ve been unable to get the observe working in the background from a standard script and have ended up with the following ruby code to make it work.

require 'sikuli.rb'
class InterruptExample
  def initialize(region=[0,0,1024,768])
    @reg=Sikuli::Region.new(*region)
  end

  def handle_interrupt(image_file)
    @reg.onAppear(image_file,self.class)
    Thread.new {@reg.observe}
  end

  def self.targetAppeared(event)
    puts(event.inspect)
  end
end

Note that the observer is at the class level, as opposed to the instance level, so it’s slightly more complicated to get access to the actual object that is observing on the event trigger. In my ruby code, I have an array of active interrupt handlers at the class level, and I query each in turn with the observable image to determine if they generated the event. Experimental code, so I won’t post it here, since I’m sure the Ruby experts amongst you will have worked out better ways of handling this.