When tests go brown

So I'm at Citcon 2010 in Wellington - and its after a morning session around BDD and Fit style testing that I'm writing this. People in the discussion use all sorts of frameworks - Cucumber, WebRat, Concordion and of course myself using easyb.

One of the interesting discussions I had last night was with Nigel Charman - part of the impetus behind the recent changes in easyb that I have done around turning it into a parse/run (instead of just run/evaluate). Nigel posited that they needed precondition checks - a third state basically that said that this test doesn't actually meet the qualifications required to run. Now you could just fail here - but that doesn't actually tell you anything, normally a failing test implies incorrect code, but in this case the code may be correct, but the data is wrong - so this mornings discussion further clarified that typically preconditions are around data being wrong or systems being unavailable. And knowing that separate from a failing test is a useful state.

As it turns out, the latest version of Junit (4.8.1) has an assume style clause [LINK] which supports the concept, but still only has the red/green colours. What I'm more interested in is reporting on it.

What does this have to do with easyb? Well, easyb also doesn't have this concept - and recently an issue was filed requesting shoulds be added to the given and when clauses (e.g. given "account is in arrears", { account =  getAccount(1234); account.inArrears().shouldBe true }. I accepted the request but said it should be in a plugin as I don't think it fit with the use of should and the concept of a failure. Now I understand why I am unhappy about it being there, and particularly I understand the pain this person is actually having.

So perhaps its time to revisit this particular issue - should easyb support another state - perhaps "brown" whereby preconditions or assumptions about the state is not met. This should be reported on to allow for dealing with in another manner.  I'll take it to the dev team and see what they think.