Wednesday, May 13, 2009

Page Based Testing With Tools Like Selenium

For a while I have been doing ad hoc runtime smoke testing of deployments with tools like Selenium. i.e. running a set of simple tests that exercise a series of pre-defined application points. The scripts where typically recorded using the excellent Selenium IDE then replayed through ant/Junit on one or more target browser platforms.

Not full regression testing but a start - right? Well these tests become a pain to maintain, and unmaintained/unreliable tests are pointless. A typical test would be something like the following :

a) login to the application
b) pick an object from an info set.
c) pass that object through several workflow steps.
d) complete the workflow.
e) log out.

The majority of the tests would be around the differences between step c) the options chosen during workflow.

I tried to take a step back and look at why these became such a pain to maintain. If I had 10 tests scripted as defined above a small change (like how I select a workflow) would effect all 10 test scripts. Multiply this to a hundred or so tests on a project with half a dozen active developers then you quickly have more broken smoke tests than working ones.

So I took a step back to try to look at a new way of writing these sort of tests so that they become less brittle. The method that seems to be working for me is to produce a more "Page" based approach rather than the test "stream" approach described above.
The concept is pretty simple, write a class that offers public methods for the types of operations that a user can perform through a browser on that page. Ensure that the operation returns another page based object.

As a simple example, lets look at a typical login page,it contains two fields userId and password. The Page object looks something like this :

public class LoginPage
{

/** Contains the actual driver we are using. */
private final Selenium selenium;

/**
* Create the page.
* @param driver the driver.
*/
public LoginPage(Selenium selenium)
{
this.selenium = selenium;

assert (selenium.getTitle().equalsIgnoreCase("My Applications Login Page));
}

/** A "good" login. i.e. one we know should allow us into the site.
*
* @param userName userId.
* @param password password to log in with.
* @return HomePage as "logged in".
*/
public HomePage performLogin(String userName, String password)
{

selenium.type("userName", userName);
selenium.type("password", password);

selenium.click("login");
selenium.waitForPageToLoad(SeleniumConstants.MAX_PAGE_LOAD_TIMEOUT);

return new HomePage(selenium);

}
}



then a simple HomePage with a search method might look like this :


public class HomePage
{

/** Contains the actual driver we are using. */
private final Selenium selenium;

/**
* Create the page.
* @param driver the driver.
*/
public HomePage(Selenium selenium)
{
super(selenium);
this.selenium = selenium;
assert (selenium.getTitle().equalsIgnoreCase("My Application Home"));
}

/** Perform a search with the supplied term.
*
* @param searchTerm term to look for.
* @return a search result page.
*/
public SearchResultPage performSearch(String searchTerm)
{

selenium.type("searchTerm", searchTerm);
selenium.click("homeSearchCommand");
selenium.waitForPageToLoad(SeleniumConstants.MAX_PAGE_LOAD_TIMEOUT);

return new SearchResultPage(selenium);

}

}


This is implemented using selenium, but the concept is applicable to any of the other tools. Note that the Home constructor contains a simple assertion that checks that the title of the page is "My Applications Home". This is important because if anything with the login fails and I stay on the login page rather than going to the home page my test will automatically terminate.

The home page again exposes methods like performSearch that again return result pages etc. This form of testing leads to tests that are *really* simple to view/read eg.

public void testPerformLoginThenSearch()
{
// these 3 lines should probably be put into a base/setup method for all tests
selenium = new DefaultSelenium("localhost", 4444, "*safari", "http://myapplication.com/");
selenium.open("/myApp/Login");
selenium.waitForPageToLoad(SeleniumConstants.MAX_PAGE_LOAD_TIMEOUT);

HomePage homePage=new HomePage(selenium);

homePage.performLogin("goul","notthatdaft!");

assertTrue( selenium.getHtmlSource().contains("Welcome back master how may I help"));

SearchResultPage searchResult=homePage.performSearch("banana hammock");
.
.
.
.
}



Something to note, if you know a test should do something different i.e. return a non-standard return page (like login fail - returning you to login) then create another public method - the page flow is the key thing here i.e. add a method like


/** A "bad" login. i.e. one we know should not allow us into the site.
*
* @param userName userId.
* @param password password to log in with.
* @return LoginPage because we are still not logged in.
*/
public LoginPage performLoginExpectingFail(String userName, String password)
{

selenium.type("userName", userName);
selenium.type("password", password);

selenium.click("login");
selenium.waitForPageToLoad(SeleniumConstants.MAX_PAGE_LOAD_TIMEOUT);

return new LoginPage(selenium);

}


Going back to the test view, we can easily hide some of the minor complexity to get started and check page content. We can easily add some utility methods to hide that ugly login and the selenium content. Lets also assume that the SearchResultPage adds methods like routeFirstObject that launches the first returned search object into a named workflow.

Our test now looks like :

public void testSearch()
{
HomePage homePage=login();
checkPageContains("Welcome back master how may I help");

SearchResultPage resultPage=homePage.performSearch("banana hammock");
checkPageContains("Hammock Object");

WorkflowRoutePage workflowRoute=resultPage.routeFirstObject("Author, Review and Translate");
.
.
.
}

This test is now amazingly simple to write, in fact you could probably convince the QA department that they can write the tests if the developers provide the page objects. Any modern IDE is going to give you strongly typed Pages for results of functions, the method names are self explanatory etc.

So how does this solve my initial problem around brittle tests?

Well lets say that how the search on the home page is launched changes (new design/ids for field etc etc.) I only have to fix the HomePage object, and fixing that single object in place means that *all* tests that use it should now run cleanly without needing changes.

I believe this approach gives the following advantages :

a) Page objects can be "grown" as tests are needed. As functionality is added to the page object, everyone using that object gets the advantage over time. i.e. the effort for new tests actually reduces as the cumulative methods are added to Pages.

b) The test scripts are very strongly typed, and are simple to understand. The page approach means the tests obviously model the flow through the application.

c) Tooling like SeleniumIDE can still be used - but take the simple steps and apply them into Page objects.

d) By making all tests that perform a login do it through the LoginPage object, you only have to change the LoginPage object if the layout/operation of login changes. i.e. fix one page object rather than a all the places calling login in scripts.

I've moved about 20 of our "smoke tests" over to this approach so far and coupled with hudson am able to run them on a series of browsers. So far so good, I'll blog further if I've been missing some huge pitfall in this approach.

Turns out this is not an original idea the good folks at WebDriver have a paper almost identical to this. Hopefully this means it is an approach that will really work!

(BTW - WebDriver looks very interesting, the HTMLUnit headless mode is great).

No comments: