Using tests to document bug fixes?

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Using tests to document bug fixes?

Postby alex.barylski » Mon Oct 04, 2010 11:41 am

I'm curious, but when you discover a bug due to a missed conditional or whatever and you add that conditional and assumingly you add the test, do you leave a comment alongside the testcase or wherever to indicate the test addressed an issue? Sort of a way of keeping precise track of bugs and issues which might not be logged in a bugtracker?

Cheers,
Alex
alex.barylski
DevNet Evangelist
 
Posts: 6259
Joined: Tue Dec 21, 2004 6:00 pm
Location: Winnipeg

Re: Using tests to document bug fixes?

Postby Eran » Mon Oct 04, 2010 11:51 am

If the issue was logged in an issue tracker or any other documentation, then I add a short comment saying something like "Fixes issue #{issue-number}". If there is no formal issue report, then I just leave it as is since there is nothing to refer to (the test should be enough to explain what it's testing against)
Last edited by Eran on Mon Oct 04, 2010 11:59 am, edited 1 time in total.
User avatar
Eran
DevNet Master
 
Posts: 3549
Joined: Fri Jan 18, 2008 1:36 am
Location: Israel, ME

Re: Using tests to document bug fixes?

Postby John Cartwright » Mon Oct 04, 2010 11:58 am

What Pytrin said, however, when ammending the tests I will usually add a comment refering to the issue number.
Code: Select all
if ($toBe || $notToBe) echo 'That is the question'; 

NEW HERE?: Please read the Forum Rules, and take the Forum Tour before posting!
User avatar
John Cartwright
Site Admin
 
Posts: 11470
Joined: Tue Dec 23, 2003 3:10 am
Location: Toronto

Re: Using tests to document bug fixes?

Postby alex.barylski » Mon Oct 04, 2010 12:51 pm

If the issue was logged in an issue tracker or any other documentation, then I add a short comment saying something like "Fixes issue #{issue-number}". If there is no formal issue report, then I just leave it as is since there is nothing to refer to (the test should be enough to explain what it's testing against)


That is what I thought was going to be the answer.

Cheers,
Alex
alex.barylski
DevNet Evangelist
 
Posts: 6259
Joined: Tue Dec 21, 2004 6:00 pm
Location: Winnipeg

Re: Using tests to document bug fixes?

Postby josh » Wed Oct 06, 2010 6:56 am

Same. I know Jenk (not try to speak for him) tries not to use any issue tracker, he uses incomplete test cases to "track" an issue, or a failing test to "track" a bug. I do use an issue tracker and sometimes I will put the bug number in a comment, but typically only if the test isn't obvious. The "bug" was more like a missing feature, I'll try to make it blend in with my other tests (no comment). If its hard to understand why that test is there, I'll put the comment so I don't decide to remove the test later when I don't recognize it.

Its almost always better to use an assertion message / good method name than a comment. Would you rather get an error about "assertion failed on line X"? Or an assertion that describes in plain English what broke? The goal is to have your assertion messages be so crystal clear on their own that you'll never need to use a debugger while refactoring code, when you edit some code and it breaks a behavior it should be crystal clear what you edited, what it broke, why it broke it, etc.. All the whos/whats/whens/ and wheres should be obvious from the test runner output alone, without requiring a developer to necessarily read the test or production code.

"Anything that is important to be in a test should be in the test. If its not important to be in a test, its important thats its not in the test" - http://www.amazon.com/xUnit-Test-Patter ... 0131495054

^ Golden rule to live by
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Using tests to document bug fixes?

Postby Jenk » Sat Oct 09, 2010 4:23 pm

Yeah, we don't use an issue tracker in my office - though we do use a story board which could be considered an issue tracker but with a subtle difference that it is feature/value driven, not bug. We don't have anything in the tests to indicate what story they are from, however we do use Acceptance tests (which directly link to the Acceptance Criteria of a given story) and Scenario style testing to make the tests readable. Such as the test class could be called WhenUpdatingAJobStatusToComplete and the test methods for JobShouldNotAcceptFurtherUpdates and JobShouldNotShowInIncompleteJobsList etc.

We found that putting "bug 1234" in a comment merely frustrated us as we'd see it, not understand it and now we'd need to open up the bugtracker to reference it. Putting atleast an explanation resolved this. Then we found the need for the bug number just disappeared and was being added on fewer and fewer occasions. Eventually it was one of the first steps to us not needing a bug tracker at all. :)

p.s. my current role is purely .NET and we use the fabulous NBehave and SpecFlow to allow us to write really meaningful acceptance tests for both us and the customer, so our tests are literally readable in the Given ... When ... Then format. :)
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: Using tests to document bug fixes?

Postby josh » Sat Oct 09, 2010 5:23 pm

That should read as ".. When ... Then" format. Not "when then format". (there is a school of thought to name your tests along the format of ['this' should happen when 'that' happens])
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Using tests to document bug fixes?

Postby Jenk » Sun Oct 10, 2010 3:26 pm

Not sure I follow what you mean, but as an example of what I mean...

Given I am an Administrator
When I create a new user with the username "Jenk"
Then I should see "Jenk" in the user list

:)
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: Using tests to document bug fixes?

Postby josh » Sun Oct 10, 2010 8:33 pm

Right, the tests names are written in the format of "when ___ then ___". As opposed to you saying "when i write a test, then I do this" (just clarifying a grammatical ambiguity)
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Using tests to document bug fixes?

Postby Jenk » Mon Oct 11, 2010 9:06 am

I'm with you now. However that's not how we do it, we treat each test class/case as a scenario, as I described. :)
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: Using tests to document bug fixes?

Postby josh » Tue Oct 12, 2010 2:06 pm

Not sure what I said wrong? I said "when then", but you use "given when then"?? Here's how I organize mine, any scenario names should go in the name of the class, not repeated in the test methods. If a test method is like whenIDoThisThatShouldHappen() that is a smell to me; instead I like to see a test method called ShouldDoThat() in a class named WhenIDoThis.

Xunit testing patterns outlines different organization methods and mentions that none is better than others, just different pros/cons. They are test case per class, test case per feature, and test case per scenario. For example Zend Framework uses test case per class. They'll have a lot of test methods in one big test case. I find that easier for someone not familiar with the tests to find something. On the other hand test case per scenario makes it hard to find where a given test might be, but makes the tests easier to manage in my opinion.

Also I think using an issue tracker is not useful for bespoke development, but is useful for "off the shelf software", in my opinion. When you're doing off the shelf software, you have to be prepared to only implement a fraction of all ideas, and its really hard to keep track of which ones are most popular among the users, etc... This is where an issue tracker helps. I agree putting the issue # into the test is worthless in almost all cases. I've done it before, when a test was really hard to understand. Obviously its wiser to just make the test easier to understand...
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Using tests to document bug fixes?

Postby Jenk » Tue Oct 12, 2010 2:35 pm

We're on the same page, apologies for the misunderstanding.

My examples of 'Given ... When ... Then ...' are for SpecFlow scenarios which are written literally as above (which then map to NBehave bindings)

Other tests unit etc are named just as you describe :)
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London


Return to Testing

Who is online

Users browsing this forum: mnlfDftfsr and 1 guest