In December of 2012, Ryan Lamontagne and I got into a good discussion on the uTest forms about whether testers should report every bug they find. We decided to kick it up a notch and debate it live in a webinar!
Tag: customer
Webinar – Three more uTest Panel webinars
It’s been a busy past few weeks. In addition to picking up two new enterprise customer accounts (uTest TTL work) I was a panelist for three more uTest webinars.
Maximizing Your Benefit From The uTest Forums
http://forums.utest.com/viewtopic.php?f=55&t=4985
Maximizing Exploratory Testing Methods
http://forums.utest.com/viewtopic.php?f=55&t=4984
How to be a Quiet Tester That Customers Shout About
http://forums.utest.com/viewtopic.php?f=55&t=4986
5 Ways to Improve Your Bug Titles
I originally posted on the uTest forum here.
Bug titles are one of the most important pieces of you bug report. They are the face of your bug, they show the its value and can help or hurt the overall efficiency of the test cycle. Far too often testers don’t give their bug titles the attention they deserve. This post will try to change that. Here are 5 tips to help you improve the titles of your bug reports.
Consider Your Audience
Like the bug report itself, the title is intended to convey information. The main difference is the title is more concise. A well written title will quickly and clearly summarize the bug and its value.
To communicate this information effectively, you need to consider your audience. Bug titles are read by different audiences who may use the title for different reasons. Testers have the difficult job of writing a title that satisfies the needs of two different audiences at the same time: The customer and your fellow testers.
Customer
When the customer or Test Team Lead (TTL) reviews the bug list, one of the first things they do is look at the title. As we talked about in Reporting High-Value Bugs – Part 2, part of reporting high-value bugs is “selling” it to the customer. The title of your bug is part of your sales pitch. Always keep the title short and to the point. You want to focus on the end result, not the actions. For example:
Use “User profile – Unable to link to Facebook” instead of “Clicking the ‘Link to Facebook’ button doesn’t do anything”
Also use words that action words that convey importance such as ‘prevented’, ‘does not’, ‘inconsistent’, ‘unexpected’ etc.
Fellow Testers
Your fellow testers use the title of your bug in a very different way. They use it to determine if the bug they found has already been reported. To help them, you need to include the key words they will be searching for.
Hopefully, before you report your bug, you search the bug list see if it has already been reported. Make a note of what you searched for because those are the words you should consider including in your title.
In Reporting High-Value Bugs – Part 2 we also talked about reporting the root cause of the bug. The same is true for the title. Your title should describe the underlying problem, not one of its many possible symptoms.
Follow the uTest Standard
uTest has a crash course dedicated to Bug Title standardization so I’m going to point you there first: http://help.utest.com/testers/crash-courses/general/bug-title-standardization
To summarize that post, every bug should be broken apart into two distinct parts. The “Area” and the “Description” The area is the place in the application where the bug occurs. The description is a brief summary of the bug. These two areas should be separated by a hyphen.
For example, in this bug title:
Homepage – The ‘Contact Us’ button is linking to the incorrect page
“Homepage” is the area and “The ‘Contact Us’ button is linking to the incorrect page” is the description
This can get a little tricky when the area is deep in the application. If there was a bug in the uTest platform on the payments screen in the Account & Settings section how should we identify that area?
In the link above, one of the authors suggests you write it like this:
Account & Settings – Payments – Total payout amount is incorrect
Personally I don’t like this suggestion. Testers who do this tend to put the navigation steps in the bug titles. That is not the place for that information. Plus having more than two sections makes the title difficult to read.
I prefer to list only the broad area of the application and include the more specific area in the description. Here is how I would write this title:
Account & Settings – The total payout amount on the Payments page is incorrect
Do Not Specify the Test Environment
Many testers include the device or environment they use to test in the title of their bugs:
[iPhone 5] User profile – Unable to link to Facebook
The landscape that we test against these days is so large that it’s no wonder that this has become more common recently. Testers feel that the device they found the bug on is an important piece of information. While that is true, the title of the bug is not the right place for it.
The main reason that this is a bad practice is because it gives false impression about the scope of the bug. Generally when testers start their bug title with the environment, they are simply stating the device that they found the bug in. But the customer may interpret that to mean that this bug is only present on that device listed in the title.
Unless you have tested against every other possible device/environment, don’t include this information in the title. It adds little value and can actually cause problems.
As with most rules, there are exceptions. Here are two:
Explicitly required
If the cycle specifically tells you to include environment information in your bug titles, you should follow the instructions.
For example, this is directly from a test cycle I recently was on
NOTE – If you find an iPad bug: Please add [iPad – iOS xx] at the beginning of you bug title.
In this situation it is perfectly fine (and even required) that you include the environment in your title.
However, you may see something like this in the instructions:
BUG TEMPLATE: Please include the following info in all your bugs: Mobile device model and OS version Description of bug Wi-Fi or 3G / 4G?
This does not mean that all this information should be in the title. It simply means that it should be specified in the body of the bug. Generally you should put this information in the ‘Specified Environments’ or ‘Additional Environment info’ fields. It is the “Bug” template, not the “Bug Title” template.
Environment specific bugs are allowed
Occasionally a cycle will allow the same bug to be reported for different environments. In this case, each one of these bugs is considered different by the customer. Since the only difference between the bugs is the environment, it is necessary to include the environment in the title. Otherwise you would have multiple bugs with the exact same title and your fellow testers would have to look at the contents of the bug to see which environments had already been reported.
Keep Consistent with Earlier Bugs
Sometimes a cycle will ask you to include some extra piece of information in the title. One example of this would be the build of the application that you tested. What I usually see happen in these situations is every tester comes up with their own way of including this information. The result is a messy bug list that looks something like this:
[b 123] Area – Description
build 123 => Area – Description
Area – Description {build 123 v.2.045.34}
123 Area – Description
This is difficult for the customer and TTL to read and makes it impossible for them to quickly scan through the list.
Assuming that the earlier bugs followed the uTest standard and everything we addressed above, you should follow the pattern established in the first few bugs. Don’t worry about being original or sticking to your own personal preference, the goal is consistency. This will make the customer’s and TTL’s jobs much easier. See how much easier this is to read?
[b 123] Area – Description
[b 123] Area – Description
[b 123] Area – Description
[b 123] Area – Description
Learn From Other Testers
You can learn quite a lot from reviewing the bug reports of your fellow testers. You can see different styles of reporting the reproduction steps, come up with new ideas of how to test, and see which types of devices are the most common.
The same can be said for the bug’s title. When you are reviewing bugs, don’t just skip over the title. Instead, take advantage of the opportunity to learn from the mistakes and successes of others.
First evaluate the title of the bug first on its own:
Does the title follow the standard? Does it include appropriate key words?
Then look at it in the context of the entire report:
Does the title accurately and efficiently summarize the bug? Does it “sell” the importance of the bug?
As you pay more attention to your own bug titles as well as the titles of other bugs, you will start to see the types of patterns we have just talked about. It will become apparent that the testers who do these types of things are the ones that are separated from the crowd. Bug titles are extremely important and should be treated that way. Keep these tips in mind and you will be one step closer to writing the perfect bug report.
I’d love to hear your thoughts on this topic. What other tips can you give your fellow testers?
Is uTest a Scam?
There are some reviews out there from testers claiming uTest is a scam and that you can’t make any money. I’ve also seen a few uTest customers complain about the quality of the testing and uTest’s sales/negotiating practices. These concerns are valid and I can understand why some people have the impression that uTest is not what it claims to be. In order to address these perceptions, we need to look at them from two points of view; The view of the customer, and the view of the tester.
The uTest customer
Complaint #1: The quality of the testing was lower than expected
I have no problem saying that there are a lot of bad testers at uTest. It’s true, there is no point in denying it. These testers only report low-value bugs, they don’t follow instructions, and generally disrupt the test cycle. Sometimes it is because they’re inexperienced testers, sometimes they’re just plain bad. Unfortunately this is one of the downsides of crowdsourcing. To uTest’s credit, they do realize this and are continuously working to improve the skills and abilities of uTesters. They also identify and remove problem testers.
On the other hand there are some absolutely awesome testers at uTest. These top testers consistently provide the customer with excellent service and high-value bugs. uTest does a pretty good job of identifying the strong testers and ensuring that they are the ones working on your projects. Keep in mind that there are literally hundreds of projects, dozens of Project Managers and thousands of testers from all over the world, so every test cycle is going to be different.
So what’s a customer to do? First you need to manage your expectations. Understand the limitations and benefits of uTest and make sure they align with your testing needs. Second, you need to be involved. Yes, uTest is a service, but the quality and success of your project is a direct result of your participation and influence.
Here is an excellent article from Elena Houser on how customers can get the most from their uTest (or any crowdsourced) testing service. It is a MUST read for any potential uTest customer: http://trancecyberiantester.blogspot.com/2012/10/crowdsourced-testing-lessons-learned.html
Complaint #2: The uTest sales process is shady
In full disclosure, I’m not a uTest customer so I’ve never gone through this process myself. However, I am a uTest TTL (Team Test Lead) and so I have worked with many different customers. I’ve seen customers who are extremely satisfied and those constantly complain. It doesn’t take long before you start to see a pattern.
I really could just do a copy/paste from above. Again, this comes down to having correct expectations and being involved. Customers who follow Elena’s advice will find that uTest’s services are well worth the money and effort. Those who don’t will be disappointed with their results.
Another point worth mentioning is that uTest is a start-up company. They have only been around a few years yet they are growing and changing incredibly fast. In just the last year I’ve seen some impressive improvements. I have no doubt that as the company matures and customer needs and expectations are better understood, the sales experience will improve and mature as well.
The uTest tester
Complaint #1: uTest is a scam
As I mentioned above, uTest is a start-up. The company grew faster than many people expected and as a result it went through some obvious growing pains. Everything about the company was (and still is) evolving. The payment process was still being worked out, bug reporting and evaluation was confusing, and in general the tester experience gave some the impression that uTest was either a scam or just unprofessional.
Admittedly, the first tester interface was terrible. It was slow, buggy, and difficult to use, which is quite ironic for a software testing company. This problem has now been addressed. uTest recently launched their new tester platform and it is so much better (read more here). Their are now several reliable ways for testers to receive payment, and there is an entire team of employees solely dedicated to the welfare of the testers. These are just a few examples of how uTest is working to improve its image and show testers that uTest is a legitimate company and a great place to work.
Complaint #2: You can’t make any money
I recently read a review from a uTester that he had reported 87 bugs but he only was paid for 16 of them. The other 71 bugs were rejected. He felt that bugs were intentionally rejected in order to avoid paying the testers. He’s not the only one to complain that testers are not adequately compensated for their efforts. Fortunately it is because of a few misconceptions.
Testers need to understand the uTest bug payment model. Customers pay uTest a set price for an agreed upon amount of work. It is up to the customer to accept or reject the bugs reported by the testers. uTest then pays the testers for the bugs (and other work) the customer accepted. Since the customer pays a flat fee no matter how many bugs are reported or accepted, they have no financial incentive to reject individual bugs. Bugs are rejected for valid reasons, not to avoid paying for them.
The other important point is testers are not paid for their efforts (there are some exceptions), they are paid primarily for the value they provide. Testers who provide the customer with high-value bugs make a lot of money. Testers who report low-value or “junk” bugs make very little money.
The bottom line is good testers can make good money working at uTest. Poor testers will be frustrated.
Conclusion
uTest is not a scam. It is a legitimate company and an amazing one at that. While uTest is not perfect, most criticisms can be answered if you look at the entire situation objectively.
Quality is Customer Value – My Quest for the uTest MVT Award
One thing I respect about the uTest management is their continual pursuit of ways increase customer value. It’s an essential business objective to ensure the health and growth of our company. ‘Value’ should be the middle name of any good tester. “Lucas Value Dargis”. Sounds pretty cool huh?
I had just finished my 26th uTest test cycle. I had put an extra amount of focus and effort into this cycle because there was something special at stake. Occasional uTest offers a MVT award which is given to the most valuable tester of the cycle. The selection process takes several things into account including the quality of the bugs found, clear documentation, participation, and of course, customer value.
The MVT award not only offered a nice monetary prize, but it’s also a way to establish yourself as a top tester within the uTest community. I decided I was going to win that MVT award.
As usual, I started by defining my test strategy. I took the selection criteria and the projects scope and instructions into account and came out with these 4 strategic objectives:
- Focus on the customer-defined ‘focus’ area
- Report only high-value bugs
- Report more bugs then anyone else
- Write detailed, easy to understand bug reports
- Be active on the project’s discussion board
When the test cycle was over I reflected on how well I’d done. I reported 9 bugs, more then anyone else in the cycle. Of those, 8 were bugs in the customer’s ‘focus’ area. The same 8 were also rated as very or extremely valuable. All the bugs were documented beautifully and I was an active participant in the discussion board.
There was no competition. No one other tester was even close. I had that MVP award in the bag. I was thinking of all the baseball cards I could buy with the extra Cheddar I’d won. I even called my mom to tell her how awesome her son was! You’ll can only imagine my surprise when the announcement was made that someone else had won the MVT award. Clearly there was some mistake right? That’s not how you spell my name!
I emailed the project manager asking for an explanation for this miscarriage of justice. The tester who won had fewer bugs, none of them were from the ‘focus’ area and they weren’t documented particularly well. How could that possibly be worth the MVT award? The PM tactfully explained that while I had done well in the cycle, the tester who won had found the 2 most valuable bugs and the customer deemed them worth the MVT award.
I was reminded that my adopted definition of quality is “Value to someone who matters” and suddenly it all fell into place. It didn’t matter how valuable I thought my bugs and reports were. It didn’t matter how much thought and effort I put into my strategy and work. At the end of the day a testers goal, his or her mission, should be to provide “someone who maters“ with the most value possible. I’m not that “someone who matters”. That “someone” is our customer.
It was a hard pill to swallow, but that lesson had a strong impact on me and it will be something I’ll carry with me moving forward. Congratulations to the MVT, I hope you enjoy all those baseball cards.