New gem: blackrat_yaml_config released

If you’ve been with me from the beginning of this blog, you’ll know I started off by building something that I called the “BorgBox”, which was a home PVR based on a Mac mini and MythTV. After moving to Boxee and being frustrated with the lack of a database, I finally (for now!) have settled on the OpenElec build of XBMC. Of course, there is scope for a lot of additional, offline tools, and since XBMC is very picky about directory structure etc. I wrote tools to aid me in moving files around. These have led to the creation of a number of small utility libraries, the first of which I’ve released as a gem on rubygems.

This first gem allows for multiple YAML configuration files to be used within a class by mixing in a YAMLConfig module and setting potential directories that the file could be included in, such as [“~/.appname”,”/etc/appname”]. This allow the configuration file to be determined with either early or late loading depending on whether the application/service needs to continue running for longer than the life of the configuration file data.

To use it, you would create code such as the following:

require 'blackrat_yaml_config'
class TestApplication
  include YAMLConfig
  config_directories :etc=>['~/.test_application","/etc/test_application"]
  config_files :global, :types
end

The relevant items are:

config_directories

This sets the search path for the configuration files. Like any search path, you start at the first, and stop as soon as you find the relevant file.

config_files

This is a list of the files that contain configuration information. As default they are postfixed with a “.yml” extension. Once they are registered, they create a methods which returns the yaml file content directly.

So if there is an entry

:database: paul.db

in the “global.yml” file, you would refer to it as:

TestApplication.global[:database]

This default allows for dynamic reloading of the configuration file. The config_files method is an alias for dynamic_config_files. There is a corresponding static_config_files which places the content into the method. So if you don’t want the flexibility of reloading the yaml file every time you refer to it, specifying

static_config_files :global, :types

will load it on the creation of the method rather than when the method is called.

I’ve heard the arguments for and against using yaml for config. To those who say “Use ruby”, I disagree. There’s more scope for using the config files across languages if they aren’t in a particular language and can be easily parsed by multiple applications. Most of my development life has been spent in mixed language environments and now is either Ruby/C/C++ or Ruby/Java, and I don’t see that changing anytime soon.

Hope you enjoy this first gem extracted from the “BorgBox” set. There are many more to come.

Installing and running MacRuby (Ruby 1.9.2) on 32-bit Intel Macs

I still use and develop on a Macbook Pro from March 2006, which has been my daily workhorse since then. I have better machines, but still find it good to use and travel almost exclusively with it, since it is the one platform I can develop and test Mac, Linux and Windows code on.

I found myself wanting to develop some Mac specific code to demonstrate a WLAN protocol communicating to a non-standard device, and decided that the traditional ruby front-end would be better served if it were Cocoa enhanced, and I’d been wanting to take a look at MacRuby for a while. Perfect opportunity I thought. Unfortunately, all of the out-of-the-box installations are targeting 64-bit environments, which my ancient Core Duo machine most certainly is not.

Rather than drop it, or switch to my quad core desktop machine, I decided to see what it would take to get it to work in a 32-bit environment, and what the requirements are for doing so.

PLEASE NOTE: This only got me to the point where I could compile
and install MacRuby and create a sample application which appears
to run. Since this is just a starting point and MacRuby fails lots
of its tests (most notably hangs during the gcd tests), don't 
blame me if your computer catches fire or refuses to work. A lot
more testing is required if this is to work reliably. As MacRuby
says in their documentation, this MAY work, but it also may not
work for you. 

First of all, check you have a recent version of XCode. Everything should compile correctly using version 3.2.1 upwards. I run 3.2.2, so I was ok. If you have an earlier version, you’ll have to login (or register) to developer.apple.com and download a later release.
2nd hurdle was LLVM. This is the core optimisation which MacRuby requires to be installed before attempting anything else. There are several issues with this, first of which is the exact version required to get it to work. After searching and finding a number of recommendations, I settled on the svn release of 0.29 with revision number 127367 as recommended in the MacRuby README documentation.
That files says: to check out and build use the following commands:

svn co https://llvm.org/svn/llvm-project/llvm/branches/release_29@127367 vm
cd vm
env CC=/usr/bin/gcc CXX=/usr/bin/g++ ./configure --enable-bindings=none
 --enable-optimized --with-llvmgccdir=/tmp
env CC=/usr/bin/gcc CXX=/usr/bin/g++ make
sudo env CC=/usr/bin/gcc CXX=/usr/bin/g++ make install

If you’ve cut and past the previous code block and it hasn’t thrown any errors, go bake a cake, or at least use another machine for a while, since this installation takes a very long time.

Once complete, you can check that the installation has succeeded by running

llvm-config --version

I’m using version MacRuby version 0.11, since that was the latest at time of writing.

Getting Sikuli Script working with Ruby(JRuby) on the Mac

Sikuli Script is one of only a handful of application that you can use to automate environments such as games, flash websites, or anything that takes a graphical approach to things. For me, getting it to support a Ruby application was key to doing what I wanted to do with it: the automation of a complex GUI.

Sikuli has a rich API and scripting language, and appears easy to use in Python, but is more complex to code to using Ruby, since it requires either JRuby or a Ruby/Java bridge and some trial and error. Since I’ve been through some of the pain in getting it working with Ruby, I’ve tried to shortcut the error for you by documenting some of my findings here.

This post looks at one approach to getting it up and running under Ruby. I tried a number of different tacks from a number of websites and found none of them to work properly. Either they were using a different version of Sikuli, or some outdated Ruby commands. This one works for me with JRuby. If you have success at getting it to work with standard Ruby using a Java bridge, please drop by and let me know.

This approach should also work for other OS’s, since the key thing is to set the CLASSPATH to the location of the jar files that Sikuli IDE installs. For the purposes of this example, I’m using the standard installation location on the Mac.

1. Grab the latest Sikuli IDE and install it from here.
2. Create a ruby file called sikuli.rb and put the following code into it

require 'rubygems'
require 'java'
$CLASSPATH << "/Applications/Sikuli-IDE.app/Contents/Resources/Java/"
#This is key to making it work under ruby. The path above is the default 
#installation location on the Mac which contains all of the
#relevant sikuli jar files.
require "sikuli-script.jar"
java_import 'org.sikuli.script.Region'
java_import 'org.sikuli.script.Screen'
java_import 'org.sikuli.script.Settings'
java_import 'org.sikuli.script.SikuliEvent'
java_import 'org.sikuli.script.SikuliScript'
include Java
Sikuli=Java::OrgSikuliScript

3. Experiment and enjoy.

There is quite a bit of documentation for Python at sikuli.org, but as we are calling functions from Ruby, you may find yourself trying a few things out in order to get them to work as you expect. For example, I’ve been unable to get the observe working in the background from a standard script and have ended up with the following ruby code to make it work.

require 'sikuli.rb'
class InterruptExample
  def initialize(region=[0,0,1024,768])
    @reg=Sikuli::Region.new(*region)
  end

  def handle_interrupt(image_file)
    @reg.onAppear(image_file,self.class)
    Thread.new {@reg.observe}
  end

  def self.targetAppeared(event)
    puts(event.inspect)
  end
end

Note that the observer is at the class level, as opposed to the instance level, so it’s slightly more complicated to get access to the actual object that is observing on the event trigger. In my ruby code, I have an array of active interrupt handlers at the class level, and I query each in turn with the observable image to determine if they generated the event. Experimental code, so I won’t post it here, since I’m sure the Ruby experts amongst you will have worked out better ways of handling this.

XML to YML conversion using ruby

I’ve switched from XML to YML as a data language for most of my code, mainly prompted by the ease of ruby/yaml. I had data for a chrononauts game in XML format. Rather than using this data as it was, I wanted to convert it to a simple YML format structure. The original code was of the form:

<chrononauts>
  <missions>
    <name>Mona Lisa Triptych</name>
    <artifact>Mona Lisa (The Real Thing)</artifact>
    <artifact>Mona Lisa (An Excellent Forgery)</artifact>
    <artifact>Mona Lisa (An Obvious Forgery)</artifact>
  </missions>
  <ids>
    <name>Squa Tront</name>
    <year>1933</year>
    <year>1950'</year>
    <year>1962'</year>
  </ids>
</chrononauts>

and I wanted it to be more like

---
chrononauts:
  missions:
    - artifact: 
      - Mona Lisa (The Real Thing)
      - Mona Lisa (An Excellent Forgery)
      - Mona Lisa (An Obvious Forgery)
      name: Mona Lisa Triptych
  ids: 
    - name: Squa Tront
      year: 
      - 1933
      - 1950'
      - 1962'

One thing in my favour was that I didn’t have any attributes to process, only data inside tags, so I figured it would be very straightforward to do using REXML and YAML. I also wanted to have hashes, arrays and simple strings where appropriate from the source data. Sprinkle in a little recursion, and the result is a pretty simple XML to YML converter. There are obvious areas for improvement, but this has worked with the data sets I have used so far, so I haven’t had any need to modity it.

#!/usr/bin/env ruby
class CardsXmlYaml
  require 'rubygems'
  require 'yaml'
  require 'rexml/document'
  YMLFile='cards.yml'
  XMLFile='cards.xml'
  
  def self.xml_process(root)
    head={}
    begin
      key=root.expanded_name
      root.children.each do |el|
        value=xml_process(el)
        if value.is_a?(String)
          next if value.gsub(/[\\n\\t\\s]/,'').empty?
        end
        if head[key].nil?
          head[key]=value
        else
          if head[key].keys.include?(value.keys[0])
            old_value=head[key][value.keys[0]]
            head[key][value.keys[0]]=[]
            head[key][value.keys[0]]<<old_value
            head[key][value.keys[0]]<<value.values[0]
            head[key][value.keys[0]].flatten!
          else
            head[key][value.keys[0]]=value.values[0]
          end
        end
      end
    rescue
      begin
        return root.value
      rescue
      end
      return nil
    end 
    head
  end
  
  def self.to_yml_file(infile=XMLFile,outfile=YMLFile,conversion_type=:file)
    output=File.new(outfile,'w')
    output.puts(YAML.dump(self.to_yml(infile,conversion_type)))
    output.close
  end
  
  def self.to_yml(doc=XMLFile,conversion_type=:file)
    begin
      doc=REXML::Document.new(File.new(doc)) if conversion_type==:file
    rescue Exception=>e
      print("Error reading xml data from file.\n")
      doc=nil
    end
    return nil if doc.nil?
    base=[]
    doc.children.each do |el|
      base << xml_process(el)
    end
    base
  end
end

Recovering from Subversion checksum error corruption

I still use subversion for all my ruby projects, despite the fact that most of the ruby world is moving to git as its main repository. I’m still at the “I’ll try it” stage (see Softies on Rails for the 4 stages of Rubyist experimentation), and will be moving over once I’ve tested that I can use Capistrano, CruiseControl, etc. with it, and know enough to support my developers/testers if and when things get sticky.
This is a log of my experiments trying to recover the files from a broken subversion repository. You may lose some data if you use this technique, and if you do, don’t blame me, but this let me recover the files, so that I could rebuild another repository as a fresh start. Note that all of the history will be lost, but if you need the files from a repository that you can’t get to because of a corruption issue.
My corrupted repository barfs at revision 51 which is a 221Mb checkin containing zip-files and other binaries using svnadmin verify gives:

svnadmin verify /svn

Output

* Verified revision 0.
.
.
* Verified revision 50.
svnadmin: Checksum mismatch while reading representation:
   expected:  589cf19ceac143315e0d61f9873ed7fb
     actual:  b0b0bf50ec4b0b089730796f3355c649

This means that something has gone wrong in revision 51. As the revision history of this isn’t that important, since this was a scratchpad project and most of the notes are of the “trying this”, “seeing what happens when” type, I wanted to see if I could pull all of the content back from it.
I’d had a look at the fsfsverify.py, but no dice. Even running it several time didn’t fix the problem with the broken revision, and doing a svn co broke in the iconex directory. However running

svn co file:///svn/trunk /tmp/trunk

did check out a large proportion of the files, up to the part where the file in the iconex directory was corrupted.
So I started to think. What would happen if I took a copy of the corrupted directory, and did an svn del on the original broken directory and checked it back in. Would that let me access the files in later revisions? I knew that none of the files in that directory had changed since revision 51, since it was purely a zip file checkin as a backup to the originals.
So

cp -r iconex iconex_backup
svn del iconex
svn ci -m 'removed broken files'
svn up

A brief wait, and the rest of the files started to check out. There were a couple of externals that needed to be changed using svn propedit svn:externals, since the connection to these files was no longer available as this repository wasn’t being served. Another svn up command, and all of the files were successfully checked out.
Hopefully this will give someone else a starting point if all attempts at recovery of the database fail, and at least you will get your non-corrupted files back.

A blog about Ruby, technology related to Ruby, useful Ruby scripts and anything else that takes Paul McKibbin's fancy.