Bug Fix Failure Alerts Using TFS 2012 Scrum Process Template

My team is using Microsoft TFS 2012 Team System Web Access with the Scrum 2.2 Process Template as our Scrum management tool. We treat bug work items similar to how we deal with PBI’s, in that we assign tasks to the bug to complete the work. Usually this results in only two tasks: One development task to fix the bug, and one testing task to verify the fix and look for any related regressions around that fix.

When the developer has completed the bug-fix task, they use the task board to move their task from the ‘In Progress’ column, to the ‘Done’ column. But what should happen when the tester determines that the bug isn’t fixed? Well, the tester moves the development task back to the ‘To Do’ column and puts a note in the history field detailing the reason why the task wasn’t done.

The problem we were having is that the developers wanted to know when their bug fixes were not complete. Usually they would find out the next morning during our daily Scrum meeting, but they wanted to know right away. I solved this problem by using the Alerts feature available in TFS 2012.

  1. From your Team System Web Access site, click the settings button (the gear icon in the top right of the page)
  2. Select the ‘Alerts’ tab
  3. In the left navigation, select the ‘Team Alerts’ section
  4. Select ‘Work Item Alerts’
  5. In the main body of the page, click on ‘New…’
  6. In the ‘SELECT NEW ALERT TEMPLATE’ window, select ‘A change is made to a work item that is assigned to me’ and click ‘OK’
  7. Give the alert a more descriptive name
  8. Add 3 new clauses: Work Item Type = Task, State Changed From Done, State Changed To To Do
  9. Click OK

You’re all set. Your developers will start receiving emails when their bug tasks are moved back to the ‘TO DO’ column.

Here’s a quick video that walks you through the above steps:
How to add a bug-fix alert in TFS

Checking vs. Testing – The Problem with Requirement Documents

The prevalent idea that testers are dependent on a requirements document to do their job is a dangerous one. Requirements are not always needed to test. In fact, in many situations, they may actually reduce a tester’s effectiveness.

The process of deriving tests directly from the requirements has several names. The ISTQB uses the term “specification-based testing”, sometimes it’s referred to as “Happy Path” testing, but I think the most appropriate name is “checking”. Michael Bolton wrote a well-known post about this topic (http://www.developsense.com/blog/2009/08/testing-vs-checking/). Checking is confirming that what we believe is actually true. Products are built in accordance with the requirements, so the requirements are what we believe to be true. When we verify that our product meets the requirements, we are “checking” the product. When a tester relies on a requirement document to test, he isn’t testing, he’s checking.

When we test, we are exploring, investigating and learning. Our actions are influenced by new questions and ideas that haven’t yet been explored. The use of requirement documents while testing can cause problems because it can give a false sense of test completeness, it can steer testers in the wrong direction, and it can reduce the independent thinking of the tester.

Gives a false sense of test completeness

If we have verified that our product meets all of the requirements, does that mean that the product has been well tested? True, you have verified that the product behaves the way we expect it to (in specific situations) but you still don’t know how the product will behave in situations not specified in the requirements.

“That wasn’t in the requirements” I’ve heard testers use this excuse many times and it drives me crazy. As a tester it is your job to investigate the product to learn about areas and behaviors outside of the requirements. A product is much more than its conformance to the requirements. It’s up to you to cover that gap between what is expected of a product and what actually is.

Checking that a product meets the requirements is necessary of course, but checking alone does not indicate test completeness.

Steers testers in the wrong direction

When you first start testing a new product or feature, what should be the first thing you test? Some might say you should verify the requirements. I challenge that view. I think the requirements should be one of the last things you test. In my opinion, a developer who writes code that didn’t meet the requirements failed to do his job. Before the test team sees the product, the developer has already spent hours working and verifying that his work is correct. Although possible, a strong developer rarely produces work that doesn’t meet the requirements. With that in mind, there is little value in retesting his work, especially when you consider the other aspects of the product that haven’t been tested yet.

Requirement documents can stifle the creativity of exploratory testing. When testers have a requirements document in front of them, they may be more likely to verify the requirements first and focus on areas where the likelihood of learning new information is the lowest. Instead, they should focus their efforts on new tests and unexplored areas where the opportunity for learning is the highest. When testing without requirements, you eliminate its influence on your testing decisions; you have to rely on your own abilities, your knowledge, and your curiosity. You test.

All testers have heard the old saying “Trust but verify”. I’m not saying that we shouldn’t verify developer’s work, only that there’s more value in performing a test for the first time, then performing the same test multiple times. Focus on the activities that produce the most value first.

Reduces independence

The idea of “independence” (See ISTQB Foundation) refers to how close a person is to a product. A developer who wrote the code would have the least amount of independence, while a person in a different company who has never seen the product would have the most independence. Independence can often be a great quality for testers. Testers with no preconceived notions of how the product is supposed to work are able to view the product with more objectivity.

Consider the common situation where a product was built according to incorrect requirements. The tester was able to verify that the product met the requirements, so was there an issue? In this case the requirements served as a false crutch to the tester. Just because the product met the requirements doesn’t mean it was working correctly. A tester with no knowledge of the requirements would have been better positioned to identify any errors because he wouldn’t have been comparing it to an incorrect “truth”.

We have now seen some reasons why testers should avoid relying on requirement documents. While they may be a necessity for different “checking” activities, testers who wish to provide value thorough “testing” must understand that their value is best realized when they test without requirements.