During long renders, I’ve been working on a little automation. For a long time I had wanted to make it easier to start up my little render ‘farm’ (just a few machines) and streamline the process of installing new versions of Houdini on them. I had long pushed this effort back using the excuse that one of the machines was still running windows which would require different logic than the other Ubuntu based machines. As the windows machine has since been migrated to Ubuntu as well this excuse was gone…
So while all my machines (including workstation) were very busy rendering, my workstation could just spare enough cycles to run a few terminal windows and a few text editor windows.
As is typical for my ‘internal’ projects, things grew a bit bigger than originally intended :). The result of this work is a collection of scripts that let me:
- boot and shutdown render machines
- switch between the latest version 10 or 11 that happens to be installed on a render machine
- retrieve the chassis status of any render machine (on or off)
- retrieve local or remote hserver status
- open a remote (through ssh) shell to a render machine
- install a version of houdini on a render machine
- publish a set of scripts to a render machine that are used by the above where needed
Some of these might seem nothing more than standard commands either in a typical Unix/Linux shell or Houdini, but using these scripts I don’t have to remember or type in any ip-address on which the command should be executed. All that’s needed to select the right machine is a number. My machines are all using incremental addresses both for the address of the machine and for the address of the IPMI (Intelligent Platform Management Interface) cards I have installed in them.
It is very easy to just string together a few of each command with incremental machine numbers to operate the command on multiple machines in one go. I’m likely to add some functionality to them which does this automatically, but as I only have a hand full of machines, it’s just as easy to use aliases for this.
Anyway, after first considering using Python as the script language, I finally decided to go with basic bash shell scripts. Bash has been my favorite Unix shell since 1991 or so. Luckily for me, both MacOSX (my workstation) and Ubuntu linux use bash as the default shell, so this works out of the box.
These scripts allow me to control the machines over the network using a combination of IPMI and ssh to connect and execute commands remotely. The IPMI is needed to allow remote booting of machines. All the other functionality is done over ssh. For this I had installed: ipmitool on my workstation and OpenSSH.
I’ve played around with the scripts quite a bit and they seem to be fairly solid. To round of this first iteration on this ‘internal’ project, I decided to do things right and add documentation and make the scripts a lot more generic. So there’s a bunch of environment variables now that specify IP address ranges, file locations, user names, etc…
For those who would have a number of render machines and would like to try out these scripts, I’ve attached a zip file with the scripts here: iscripts
Installing the scripts is a matter of unzipping it in a location you want. You should have an environment variable set to point to the location of the iscripts folder:
I’ve included below the environment variables and aliases that I’ve setup to make things as easy as possible to use. The RS* variables are required to make the scripts work. The first two commands (‘source .h*ver’) include the contents of the files .h10ver and .h11ver.
In my case these currently contain:
These two small files have to be updated when I install a new version of Houdini on my workstation and are only used in the alias definitions below (they are not used for the scripts). The workstation install of Houdini is currently still a separate step and is not supported by the scripts.
Anyway, it was an interesting little project, which makes things a little more efficient. If you’re interested in using these scripts or want to discuss them please don’t hesitate to reply to this post or send me an email. I would be interested in sharing thoughts on pipeline stuff like this and to discuss/explain more about these scripts.
Below follows the variable part of my local (workstation) .profile file (can be found (or created) in your home folder).
# render farm vars
# setup Houdini version aliases (including env settings)
alias h10setup='oldpath=`pwd`;cd /Library/Frameworks/Houdini.framework/Versions/$H10VER/Resources;source houdini_setup;cd $oldpath'
alias hs10='h10setup;hserver -q;sleep 1;hserver'
alias h11setup='oldpath=`pwd`;cd /Library/Frameworks/Houdini.framework/Versions/$H11VER/Resources;source houdini_setup;cd $oldpath'
alias hs11='h11setup;hserver -q;sleep 1;hserver'
alias cdhou="cd $H"
# setup default hooding environment