After the grand experiment described below I ported a large number of the smoke tests over, these happily ran on my Mac under both Safari and Firefox 3.
Of course despite my bias for OSX/Linux apparently one or two internet users still use IE. So we probably need to test on that "other" platform too.
To do this I setup three VirtualBox instances, running IE6/7/8 respectively. I then modified our base test to take a set of parameters for test server/port/browser. i.e. make our base test setup the driver as follows :
selenium = new DefaultSelenium(seleniumClientIP, seleniumClientPort, seleniumClientBrowser, baseTestURI);
Then I created a quick ant script to perform all my smoke test on each combination, wrapped it up in a Hudson instance to run it periodically and report any JUnit failures.
All sounds rosy doesn't it?
Whilst the tests all run fine in Safari and Firefox on any of the hosts/virtual machines. It appeared that we are not so lucky with IE. Bottom line was that without intervention tests running in the IE VMs fail about 15% of the time with "Internet explorer has encountered a problem". This modal dialog eventually leads to timeouts in the tests, and the windows/IE instances don't go away without manual intervention. As previously discussed if the tests are flakey or out of date they become an irrelevent distraction.
A lot of internet digging this weekend has led to this thread. This has two salient bits of advice :
a) Even though the IE selenium runs are much slower that those in Safari/Firefox, it can still get into trouble with timing. setting the default selenium speed like this selenium.setSpeed("500") seems to help.
b) If the above doesn't fix all issues, turn off windows error reporting to stop the instances getting hit with the modal dialog. (The tests will still fail, but the instance is recoverable without manual intervention).
Since I put these changes in place, we have not had a single IE fail. Time will tell of course but things certainly look positive so far!