Using tests to document bug fixes?

[alex.barylski]

I'm curious, but when you discover a bug due to a missed conditional or whatever and you add that conditional and assumingly you add the test, do you leave a comment alongside the testcase or wherever to indicate the test addressed an issue? Sort of a way of keeping precise track of bugs and issues which might not be logged in a bugtracker?

Cheers,
Alex

[Eran]

If the issue was logged in an issue tracker or any other documentation, then I add a short comment saying something like "Fixes issue #{issue-number}". If there is no formal issue report, then I just leave it as is since there is nothing to refer to (the test should be enough to explain what it's testing against)

[John Cartwright]

What Pytrin said, however, when ammending the tests I will usually add a comment refering to the issue number.

[alex.barylski]

If the issue was logged in an issue tracker or any other documentation, then I add a short comment saying something like "Fixes issue #{issue-number}". If there is no formal issue report, then I just leave it as is since there is nothing to refer to (the test should be enough to explain what it's testing against)

That is what I thought was going to be the answer.

Cheers,
Alex

[josh]

Same. I know Jenk (not try to speak for him) tries not to use any issue tracker, he uses incomplete test cases to "track" an issue, or a failing test to "track" a bug. I do use an issue tracker and sometimes I will put the bug number in a comment, but typically only if the test isn't obvious. The "bug" was more like a missing feature, I'll try to make it blend in with my other tests (no comment). If its hard to understand why that test is there, I'll put the comment so I don't decide to remove the test later when I don't recognize it.

Its almost always better to use an assertion message / good method name than a comment. Would you rather get an error about "assertion failed on line X"? Or an assertion that describes in plain English what broke? The goal is to have your assertion messages be so crystal clear on their own that you'll never need to use a debugger while refactoring code, when you edit some code and it breaks a behavior it should be crystal clear what you edited, what it broke, why it broke it, etc.. All the whos/whats/whens/ and wheres should be obvious from the test runner output alone, without requiring a developer to necessarily read the test or production code.

"Anything that is important to be in a test should be in the test. If its not important to be in a test, its important thats its not in the test" - http://www.amazon.com/xUnit-Test-Patter ... 0131495054

^ Golden rule to live by

[Jenk]

Yeah, we don't use an issue tracker in my office - though we do use a story board which could be considered an issue tracker but with a subtle difference that it is feature/value driven, not bug. We don't have anything in the tests to indicate what story they are from, however we do use Acceptance tests (which directly link to the Acceptance Criteria of a given story) and Scenario style testing to make the tests readable. Such as the test class could be called WhenUpdatingAJobStatusToComplete and the test methods for JobShouldNotAcceptFurtherUpdates and JobShouldNotShowInIncompleteJobsList etc.

We found that putting "bug 1234" in a comment merely frustrated us as we'd see it, not understand it and now we'd need to open up the bugtracker to reference it. Putting atleast an explanation resolved this. Then we found the need for the bug number just disappeared and was being added on fewer and fewer occasions. Eventually it was one of the first steps to us not needing a bug tracker at all.

p.s. my current role is purely .NET and we use the fabulous NBehave and SpecFlow to allow us to write really meaningful acceptance tests for both us and the customer, so our tests are literally readable in the Given ... When ... Then format.

[josh]

That should read as ".. When ... Then" format. Not "when then format". (there is a school of thought to name your tests along the format of ['this' should happen when 'that' happens])

[Jenk]

Not sure I follow what you mean, but as an example of what I mean...

Given I am an Administrator
When I create a new user with the username "Jenk"
Then I should see "Jenk" in the user list

[josh]

Right, the tests names are written in the format of "when ___ then ___". As opposed to you saying "when i write a test, then I do this" (just clarifying a grammatical ambiguity)

[Jenk]

I'm with you now. However that's not how we do it, we treat each test class/case as a scenario, as I described.

[josh]

Not sure what I said wrong? I said "when then", but you use "given when then"?? Here's how I organize mine, any scenario names should go in the name of the class, not repeated in the test methods. If a test method is like whenIDoThisThatShouldHappen() that is a smell to me; instead I like to see a test method called ShouldDoThat() in a class named WhenIDoThis.

Xunit testing patterns outlines different organization methods and mentions that none is better than others, just different pros/cons. They are test case per class, test case per feature, and test case per scenario. For example Zend Framework uses test case per class. They'll have a lot of test methods in one big test case. I find that easier for someone not familiar with the tests to find something. On the other hand test case per scenario makes it hard to find where a given test might be, but makes the tests easier to manage in my opinion.

Also I think using an issue tracker is not useful for bespoke development, but is useful for "off the shelf software", in my opinion. When you're doing off the shelf software, you have to be prepared to only implement a fraction of all ideas, and its really hard to keep track of which ones are most popular among the users, etc... This is where an issue tracker helps. I agree putting the issue # into the test is worthless in almost all cases. I've done it before, when a test was really hard to understand. Obviously its wiser to just make the test easier to understand...

[Jenk]

We're on the same page, apologies for the misunderstanding.

My examples of 'Given ... When ... Then ...' are for SpecFlow scenarios which are written literally as above (which then map to NBehave bindings)

Other tests unit etc are named just as you describe