For whatever reason, I'm having a hunt through the TestNG source code. It's got ".classpath" and ".project" files, which is generally an indicator that the project can be imported into eclipse. Sure enough, it can, but there's a missing variable that needs to be set up: TOOLS15_LIB should point to your JDK's "tools.jar" file.

I wonder how that works on OS X, where certainly the older JDK installations didn't have a "tools.jar" to point at.

The problem with acceptance and regression testing is that the errors coming back from an automated build don't really help identify the problem. A stacktrace letting you know that some element wasn't present on the page doesn't necessarily tell you why that is. However, the stacktrace dumped out to the screen by the app server (or the wedged UI of your desktop app, or... you get the idea) tells you an awful lot.

Wouldn't it be nice to capture a screen-shot when a test fails? And did you know that it's ridiculously simple to do this using bog-standard Java? The code snippet (without error checking) you need goes something like:

public void grabScreenShot(String testName) {
    File dir = System.getProperty("place.to.dump.images", "/tmp/build");
    Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
    BufferedImage image = new Robot().createScreenCapture(new Rectangle(screenSize));
    ImageIO.write(image, "jpeg", new File(dir + "/" + testName + ".jpeg"));
}

If you're using JUnit 3, just set up a new TestListener that does this when a test fails or has an error. If you're using some other test framework then you'll need to figure out what to do. How hard can it be?

Now, I know that I'm a little late to the party, but I'm on my first Java 5 project and I'm liking it. Given the shiny goodness in it, from Doug Lea's concurrency library through generics and covariant return types to the extra insight that can be gained through using JMX on the virtual machine, there's not much reason not to use it. Even WebSphere, notoriously slow as it is to pick up on new-fangled goodness in Java, has finally caught up.

Now, if only I had to chance to explore those further reaches of the API....

One of the great things about programming in Java is the number of abstractions available to you. These let you think about a problem in a very specific way depending on the layer of the application you happen to be sitting in. Hibernate, for example, let's you think in terms of Objects rather than database tables (though, obviously, as you progress you need to be aware that it is just an ORM tool and you can't hide from the "R" forever) Spring's Inversion of Control container allows you focus exclusively on the relationships between things, and then ignore it for the rest of your coding. And so on.

Of course, there is a down-side, as the diagram that Tim Bray linked to demonstrates. That's an awfully huge stack trace....

We've got a problem. The problem is that there's this library that we have to use which needs to be configured using something other than code. Through an XML file, perhaps, or an LDAP schema. This is fine if the configuration is set in stone, but it's not. For the sake of efficient integration testing you want to be able to replace chunks of the system's configuration between tests, or perhaps you want to only set up a subset of the configuration. What can you do?

Let's take a step back and think about the various points at which we can intervene in this system:

• By using the normal configuration mechanisms, making the library re-read these with every test
• By somehow subverting the configuration mechanism, perhaps by noticing that certain system properties are used for default values
• By plugging in a new configuration mechanism which can be manipulated in code
• By inheriting from key classes in the library and configuring it to use those instead
• By inheriting from key classes in the library and forcefully injecting them using reflection
• By re-writing part of the library and submitting those changes back to the main development trunk
• By taking advantage of class-loader ordering, and placing overridden but binary compatible version of key library classes in an earlier class-laoder (that is, the class has the same public interface, name and package as one in the library)
• By the "nuclear option" of using AOP and wrapping any offending pain points

I'm sure that this list isn't complete (doubtless I'll wake up at 4 in the morning with another bright idea) but it's enough to get us going.

The interesting thing about this list is that the amount of political resistance tends to increase as you head further down it --- it tends to be easier to get acceptance of the use of "official" configuration mechanisms or extension points than it is to get buy-in for minor tweaks to the library. This in turn is easier than trying to persuade people that the use of AOP is a sane course of action. The use of AOP can be a lot like trying to trim your nails with a chainsaw: you can do it, but it's more than likely going to end up in a bloody mess unless you're very careful.

So, we have our list of intervention points. How to use them?

My preference would tend to be something that's obvious and which fits into the overall philosophy of the library. As such, providing a new configuration mechanism through an existing extension point or contributing a change to the library back would be the avenues that I'd want to explore first. Constantly spooling files to disk seems a little distasteful to me, and sometimes libraries are written in such a way as to prevent reloading once they're up and running. Additionally, the reading of config files can take some time, as anyone who's tried it with Hibernate can probably tell you, so it's not an ideal solution.

Once those possibilities have been exhausted, my Perlish tendencies tend to show and I start to reach for reflection. If the library is lousy with final classes, then AOP or class-loader abuse starts to look like a decent option. I'd want to step outside and get some air before pressing on with that plan....

Where am I going with this? Good question. I'm not sure, but I've been thinking about the various points at which a developer can intervene in a system and thought it might be nice to share. What do you like to do? How do you deal with these issues?

Ths is good news! WebWork 2.2 is out. The big thing that I like about it is that there's now native support for Spring rolled into the core distribution. I've been using the Spring/WebWork integration since it first appeared and it's pretty slick --- try it, you might like it. Obviously, there's a bunch of other Good Stuff in there too.

This is also the last release of WebWork under the aegis of the OpenSymphony group. After this, we'll see a migration to the Struts Action Framework. As a long time WebWork user, I find this new direction interesting. On theone hand, it makes it far easier to introduce a kitten-friendly webapp framework into clients who insist that Struts is the only answer (for whatever reason) On the other hand, WebWork will now acquire an inertia which will make it harder to address some of the underlying issues with WebWork. In addition, it'll be interesting to see if the "best practices" of the WebWork community infect the Struts community or the other way round. We'll see.

A couple more thoughts about the TestEnvironments that I wrote about previously. Most of my feelings and thoughts about them is derived from Domain Driven Design's ubiquitous language.

When talking to a developer or a QA they tend not to talk about "the acceptance test environment where the web server is bound to port 8080 and the database should be in-memory and contain a certain amount of test data" They tend to talk about the "acceptance test environment" or the "functional test environment", and by using those labels they know that one has an appserver running remotely and a database that's pre-loaded with test data and the other one is just an in-memory database. As such, I like my code to talk about the AcceptanceTestEnvironment and the FunctionalTestEnvironment.

Which leads on to another thing that's been buzzing around my head, and that's the fact that there's normally at least two domain models that are extant within a code-base; one in the application itself, and one in the tests that surround and support this code. The interesting thing is that quite often the same domain concept, such as User or Account, gets referred to in both places. Does this mean that the same classes representing these concepts be used in both places? I'd suggest not. Although they both happen to have the same name, they typically refer to completely different things. One refers to the abstract concept within the application, and this has radically different roles and responsibilities to the other, the on-screen representation that's used within the acceptance test code.

The logical extension of this is that the test code should have its own ubiquitous language that may mirror that of the application but may be very different. This means that testing code, which sometimes doesn't get the TLC that it deserves, needs to be approached with the same sort of rigour that production code should be approached with.

Well, that's what I think anyway. I should really get comments up and running shouldn't I?

Yesterday I wrote about the moving parts that allow you to write acceptance tests for a Java webapp without going insane. We talked about XPath and how wonderful it is, as well as the roles and responsibilities of the various classes within the system. I also mentioned that I was undecided about what to blog about because there were two things that I really liked and wanted to tell you, my dear reader, all about. Well, now it's time for this mysterious second thing.

I can tell that you're on tenterhooks. It's why I blog so often.

Let's take a step back from our webapp and think for a second. We want the acceptance tests to be fast, but we'd also like to be able to run them against a stack as similar to that used in production as possible. Unless you happen to be deploying to Jetty it's likely that deployment takes the better part of 5-10 minutes and dozens of mouse clicks (yes, WebSphere, I'm looking at you) Our two goals appear to be mutually exclusive. And what about running our integration tests? You know, the ones that check that we're using Hibernate correctly. At the other end of the scale, those pesky QA people will probably have their own test environments that they'll want to use, and we'd like to be able to run our automated acceptance tests against those too.

Hang on! "Test environments"? There's a pattern in there somewhere (bear with me, I've not written one of these before)....

Automated tests could be run in a variety of test environments

Therefore, create a class to represent each expected test environment. This class can be used to obtain information about resources used or presented by the application, such as DataSources or the base URL of a webapp.

Ideally, a TestEnvironment should be created by calling the default constructor, and before being created a check should be performed to see whether or not another TestEnvironment is in use:

private TestEnvironment setUpEnvironment() throws Exception {
    if (TestEnvironment.isEnvironmentAlreadySetUp()) {
        return TestEnvironment.getCurrentEnvironment();
    }
    return new AcceptanceTestEnvironment();
}

The smarter amongst you will probably tell me not to use statics for this sort of thing, and will suggest a better solution. I can live with a bit of constructive criticism.

As an aside, if a particular environment doesn't require a resource (such as an app server in your integration tests) then I've found it best for the TestEnvironment to throw an exception rather than simply returning null. YMMV. You'll also find that instantiating the TestEnvironment in a TestSetup decorator or in a TestSuite is a Good Thing.

Previously, I'd posted about testing a webapp We'd started at the top level, looking at how we could write our acceptance tests. This is all very well, but what's sitting underneath that? I'm not quite sure where to start here because there are two things that I've seen that I really like. Let's do the one that saves the most lines of code, and then, in another post, the other one.

Our application appears to the user as a series of pages. It makes sense that the acceptance tests should therefore query Page Objects about what it expects to see. Each Page Object represents a single page within your system and might be composed of smaller elements, such as a NavigationMenu. Personally, I like to query the Page Objects using objects representing what I expect to see on the page, other people like to simply use strings. Either way is fine, but the trick is to avoid exposing the raw structure of the page to the asking object, therefore:

accountPage.isAccountSummaryDisplayedFor("Barry");

is infinitely preferable to:

Iterator i = accountPage.getAccountSummaryTable().rows().iterator();
...

The Page Object relies on an interface wrapping our browser implementation (WebDriver and Browser are both names that I've seen) that exposes the current DOM which can then be queried using XPath. Because you're simply exposing the DOM and providing a few simple methods to submit forms and click links, the WebDriver interface can be pretty slim, reducing the pain of writing new implementations. In order for this to work, your webapp should be generating valid XHTML. At the very least, it should be returning well-formed XML.

There are several reasons to use XPath, but the most important ones are that it helps reduce the lines of code required to write tests, improving the maintainability of the tests, it's easy to read (especially if you stick with a limited subset of XPath), and it means that there aren't dozens of barely used classes cluttering your system ("Table", "TableRow", "TableCell", *shudder*) In addition, because these cluttering classes don't exist, you're far less tempted to break the encapsulation of the Page Objects. Hurrah!

Let's now think about the roles and responsibilities of the various layers we've just introduced. The acceptance tests are the only places where you should be making assertions (which also makes it clearer exactly where a test is failing because you don't need to dig through a stack trace). This means that the deeper levels should simply provide boolean methods for testing the (lack of) presence of constructs within the page. The acceptance tests have no idea of the structure of the page. This knowledge is held within the Page Objects, which know how to clickOnLinkForAccount("foo") and tell you isRecordDisplayedFor(barry). The Page Objects use XPath to query the DOM, which is obtained from the WebDriver. The WebDriver knows how to communicate with the outside world, how to click on links and submit forms.

So, looking back, the stack looks something like:

Acceptance Test makes assertions based on boolean methods on Page Objects, which use XPath to query the DOM returned by the WebDriver.

BTW, If you're interested, Mike Roberts has a post about using wikis to store acceptance tests.

It's time to talk testing. Specifically, it's time to talk about a strategy for testing java web apps. Let's take it from the top, from the level of the person writing the initial tests, down to the place where the developers live. Let's not rush ourselves; we've got a while so more than one part is okay.

At the top, there's one of two types of acceptance test. The initial type is a normal looking JUnit test, with several "testFoo" methods. The difference is that these tests are written in the language of the business:

public void testNavigationLinksShouldMaintainTheQueryUsedToGenerateTheResultList() throws Exception {
        loginAs("Barry White");
        selectDJ("Dave Lee Travis");
        searchForMyRecordsOwnedByTheDJStartingWith("t");
        countNumberOfPagesOfResults();
        gotoNextPage();
        countNumberOfPagesOfResultsAgain();
        verifyThatTheNumberOfPagesSeenWasTheSameBothTimes();
}

Even my mum could read that. These are written by the QAs using an IDE that generates stub methods so that the thing compiles cleanly. The cleanly compiling tests are then checked in to our source control system and run as part of our normal build.

But wait! Surely those tests won't pass, and will therefore cause the build to fail horribly? Not if you use a TestSuite that supports allowing a test to fail until the first time it's passed, and then never allows it to fail again. You could even produce a nice summary page in your Continuous Integration environment to allow the business to track the progress of the project.

The other type of test that we run is a FIT test. These can work really well, and they can also bring an entire project to its knees; you just need to know what's good and what's not. What's not include "clicky" tests, that interact directly and extensively with the UI. These swiftly become very brittle and require a lot of TLC to work properly as the GUI changes. What works well? I'm still exploring, but something nice and high level, that deals with the intent is clearly the way forwards. Something like

| login as | Barry White |

rather than

| goto |  home url |
| click | button with id | login |
| select element | username |
| enter | Barry White |
| select element | password |
| enter | secret password |

One of those is clearly a suboptimal way of doing things. Guess which!

So, we've written our high level acceptance tests. What next? You'll just have to wait....