Tag: bug report

How to Write a Quality Bug Report

I worked with Aaron Weintrob to put together a uTu course on how to write quality bug reports. The intended audience of this course is new uTesters with little or no testing experience. We kept it relatively short and simply highlighted the key areas. We may create additional courses for various sections where we can dig in a little deeper. Here is a link to the course. Below is the course’s content:

INTRODUCTION

Writing a good bug report is one of the most talked-about topics in the testing world. The art of creating a well-written bug report requires a balanced combination of testing and communication skills. This course provides advice and tips geared towards helping you create bug reports that are informative and actionable, thus improving their value to the customer.

WHAT MAKES A GOOD BUG REPORT?

Most testers understand the role of a bug report is to provide information, however a “good” or valuable bug report takes that a step further and provides useful information in an efficient way.

To help us get started writing valuable bug reports, we are going to focus on a few key areas:

  • The Title
  • Actions Performed (Steps)
  • Expected and Actual Results
  • Attachments

THE TITLE – THE GOOD, THE BAD, THE UGLY!

The title is the face of your report. It’s the first thing anyone sees and it’s importance cannot be overstated. A good title helps reduce duplicate issues and can quickly convey a summary of the bug.

It’s best to avoid generic problems in the title. For example, these should never be used:

  • XYX is not working properly
  • Issue with XYZ
  • XYZ is corrupted/does not look right

The above example titles add little value in describing the problem. By nature, every report is describing something that is not working as it should. Be specific about what makes it “not working.”

Instead of: Sorting is not working properly.
Try: Sorting is happening in reverse order.

Instead of: Issues with GUI on navigation bar.
Try: Navigation bar is wrapping to a second line.

Often times, bugs are migrated into the developer’s database that may contain hundreds, if not thousands, of other issues. Imagine trying to search this database for “navigation bar”. That search will return every issue related to the navigation bar. Searching for “wrapping to second line” is much more specific making it easier to find the bug. Your bug report needs to survive (and be useful) beyond the current test cycle; a strong title will help it through it’s journey.

ACTIONS PERFORMED – ADVICE FOR EXPLAINING YOUR STEPS

This is the body of your report. The goal of this section is to show the reader how to reproduce the bug. Since this area usually contains the most information, it’s important to keep it concise and easy to read. Always number your steps and kept them short and to the point.

Tip: Using a prerequisite can reduce the number of steps.
Instead of listing out every step to login in, start your steps with: “Prerequisite: User is logged in”

Tip: Find the direct path to the bug
Often times, testers will stop at the point where they found a bug and log their last few actions. However, the most helpful bug reports are those that distill the report down to the core reproduction steps.

It’s a good exercise to reproduce the bug by following the steps you’ve just outlined. This will help ensure you’ve included everything the customer will need to reproduce it as well.

Sometimes digging a little deeper below the surface of the bug can add additional value. Here are some examples of how adding a bit more effort or thought will produce a higher quality report.

Example 1: Provide additional useful information
Scenario: You find that a video does not play.
Good: Mention if it happened on all videos and not just the one mentioned in report.
Better: Specify if the issue is reproducible on more than one browser or device.
Best: Upload a speed test showing that bandwidth was adequate when testing was happening.
Lesson: Try to identify and answer follow-up questions before the customer asks them.

Example 2: Report the bug, not a symptom of the bug
Scenario: We are testing an Address input field. We find that the Address field allows “1234567890″ and it also allows “!@#$%^&*()_+”
Lesson: These are two different symptoms of the same bug. Closer inspection would reveal that the real issue is the Address field isn’t being validated at all. The problem may be more serious than the first symptom you find.

EXPECTED AND ACTUAL RESULTS – WOULDA, COULDA, SHOULDA

Now that you have described how to reproduce the bug, you need to explain the problem and the desired behavior.

Tip: When describing expected results, explain what should happen, not what shouldn’t happen.
Instead of: The app shouldn’t crash.
Try: The user is taken to XYZ screen.

Tip: When describing actual results, describe what did happen, not what didn’t happen.
Instead of: The user wasn’t taken to the XYZ screen.
Try: The user remained on the ABC screen.

ATTACHMENTS – WHAT TO DO AND WHAT NOT TO DO

Attachments add to the bug’s value by offering proof of the bug’s existence, enabling the customer to reproduce it or helping the developer fix it. Each attachment should add to the value of the bug in at least one of these three ways.

The following are some tips and guidelines to keep in mind when adding attachments:

IMAGES

  • Adding images is a quick way to add context to your bug. Consider adding an image even if you also have a video.
  • Highlight the area(s) of interest in your image.
  • Attach the image files directly to the report. Don’t put images in a Word document or a zip file.
  • Use images to illustrate static issues.

VIDEOS

  • Video confirms your steps were accurate at the time the issue was created. For example, a screen grab of an error message isn’t as useful as seeing what went into creating that error message.
  • Actions in the video should match the steps listed in the bug report.
  • Videos should be trimmed to only show the bug.
  • Provide video if the steps are complex.
  • External/live videos can be more impactful than mirrored videos because you can see hand gestures or you touching a button on the screen.

LOG FILES AND OTHER TIPS

Avoid proprietary file types (like .docx). Use .txt instead.
Avoid compressed (.zip) files unless specifically asked for or approved by the TTL, PM, or customer.

UTEST ETIQUETTE – GENERAL OVERVIEW OF PROPER BEHAVIOR

It is important to remember that you are representing the TTL, the test team, and all of uTest when you work on the test cycle. Your fellow testers rely on you to write a good title for your bug report so they won’t file a duplicate bug. TTLs depend on clean, good reports to ensure the customer receives value from the cycle. uTest needs quality work from everyone so we can continue to work in the field we all love.

ADDITIONAL READING

Here are some valuable discussions about bug reports from the uTest Forums:

Two contributions to the uTest University

Back in December of 2013, uTest officially launched the uTest University (blog post) which is intended to be a single source for testers of all experience levels to access free training resources. This is a neat opportunity for testers to contribute to the growth and development of the testing community by creating courses and writing articles. The university also offers  Author Page

My first course was derived from a uTest forum’s post I I wrote back in June of 2013. I was on a cycle where the customer required logs from Charles Web Debugging Proxy be attached to every bug report, but none of the testers (myself included) or the knew what that was or how to use it. I spent some time learning how to use the tool and then put together a tutorial to share with the rest of the team. Fast forward 8 months later several other customers started required the same thing. To make the information a bit easier to find the tutorial was turned into a uTu (uTest university) course:
How to Set Up Charles Web Debugging Proxy for iOS Devices and Windows 8

My second course came at the request of the uTest Community Management team. They needed a tutorial for new testers to show them how to create videos (screencasts) of their bugs. They specifically wanted it based around the free tool Screencast-O-Matic. I had actually never used that tool before, so I spent some time getting familiar with the tool. I also compiled a list of suggestions and tips based on things I see frequently in the videos of other testers. The result is:
How to Set Up and Use Screencast-O-Matic

 

Webinar – Should testers report every bug they find?

In December of 2012, Ryan Lamontagne and I got into a good discussion on the uTest forms about whether testers should report every bug they find. We decided to kick it up a notch and debate it live in a webinar!

http://forums.utest.com/viewtopic.php?f=13&t=4430

Webinar – Finding bugs in mobile devices

I was able to join Kayla Cox and Todd Smith for a uTest webinar to talk about testing mobile devices and how to find high-value bugs.

Since my microphone was terrible (and I might have been mumbling a little ) here is a summary of the points I made in our discussion.

Crashes
Understand that not all crashes are valuable.
Out of memory crash may be due to other apps using up 90% of your memory and the app you are testing just pushed you over the limit. The best way to know for sure, is to have a clean test bed. Restart your phone after you install a new app, and make sure no other apps are running in the background.

When you do get a memory related crash, use a memory management app to help you see where your memory usage spikes. Being able to identify a reproducible memory crash is usually a high-value bug

Connection Issues

  • Kill your connection while data is being transferred
  • Unplug your wi-fi router/modem
  • Turn on airplane mode
  • Turn of wi-fi on your device
  • Turn off cellular data on your device
  • Find places near you that have low or no signal and test there

Interaction with native and popular apps

  • Share something via email with no email set up
  • Log in using Facebook account with/without the Facebook app installed
  • Interrupt testing with phone calls, text messages, FaceTime calls etc
  • If the app changes the phone settings, make sure it does it correctly. Change it back manually in settings and see how the app responds

Investigation and Documentation
There are many topics on how to write good bug reports but there are a few points worth reiterating

  • Provide exact reproduction steps
  • Do root cause analysis – don’t report symptoms. I once saw 3 testers reported 3 different symptoms of the same bug. On the surface they all looked like different bugs, but a little analysis showed they were all caused by the same step they all overlooked. 

5 Ways to Improve Your Bug Titles

I originally posted on the uTest forum here.

Bug titles are one of the most important pieces of you bug report. They are the face of your bug, they show the its value and can help or hurt the overall efficiency of the test cycle. Far too often testers don’t give their bug titles the attention they deserve. This post will try to change that. Here are 5 tips to help you improve the titles of your bug reports.

Consider Your Audience

Like the bug report itself, the title is intended to convey information. The main difference is the title is more concise. A well written title will quickly and clearly summarize the bug and its value.

To communicate this information effectively, you need to consider your audience. Bug titles are read by different audiences who may use the title for different reasons. Testers have the difficult job of writing a title that satisfies the needs of two different audiences at the same time: The customer and your fellow testers.

Customer
When the customer or Test Team Lead (TTL) reviews the bug list, one of the first things they do is look at the title. As we talked about in Reporting High-Value Bugs – Part 2, part of reporting high-value bugs is “selling” it to the customer. The title of your bug is part of your sales pitch. Always keep the title short and to the point. You want to focus on the end result, not the actions. For example:

Use “User profile – Unable to link to Facebook” instead of “Clicking the ‘Link to Facebook’ button doesn’t do anything”

Also use words that action words that convey importance such as ‘prevented’, ‘does not’, ‘inconsistent’, ‘unexpected’ etc.

Fellow Testers
Your fellow testers use the title of your bug in a very different way. They use it to determine if the bug they found has already been reported. To help them, you need to include the key words they will be searching for.

Hopefully, before you report your bug, you search the bug list see if it has already been reported. Make a note of what you searched for because those are the words you should consider including in your title.

In Reporting High-Value Bugs – Part 2 we also talked about reporting the root cause of the bug. The same is true for the title. Your title should describe the underlying problem, not one of its many possible symptoms.

Follow the uTest Standard

uTest has a crash course dedicated to Bug Title standardization so I’m going to point you there first: http://help.utest.com/testers/crash-courses/general/bug-title-standardization

To summarize that post, every bug should be broken apart into two distinct parts. The “Area” and the “Description” The area is the place in the application where the bug occurs. The description is a brief summary of the bug. These two areas should be separated by a hyphen.

For example, in this bug title:
Homepage – The ‘Contact Us’ button is linking to the incorrect page
“Homepage” is the area and “The ‘Contact Us’ button is linking to the incorrect page” is the description

This can get a little tricky when the area is deep in the application. If there was a bug in the uTest platform on the payments screen in the Account & Settings section how should we identify that area?

In the link above, one of the authors suggests you write it like this:
Account & Settings – Payments – Total payout amount is incorrect

Personally I don’t like this suggestion. Testers who do this tend to put the navigation steps in the bug titles. That is not the place for that information. Plus having more than two sections makes the title difficult to read.

I prefer to list only the broad area of the application and include the more specific area in the description. Here is how I would write this title:
Account & Settings – The total payout amount on the Payments page is incorrect

Do Not Specify the Test Environment

Many testers include the device or environment they use to test in the title of their bugs:
[iPhone 5] User profile – Unable to link to Facebook

The landscape that we test against these days is so large that it’s no wonder that this has become more common recently. Testers feel that the device they found the bug on is an important piece of information. While that is true, the title of the bug is not the right place for it.

The main reason that this is a bad practice is because it gives false impression about the scope of the bug. Generally when testers start their bug title with the environment, they are simply stating the device that they found the bug in. But the customer may interpret that to mean that this bug is only present on that device listed in the title.

Unless you have tested against every other possible device/environment, don’t include this information in the title. It adds little value and can actually cause problems.

As with most rules, there are exceptions. Here are two:

Explicitly required
If the cycle specifically tells you to include environment information in your bug titles, you should follow the instructions.

For example, this is directly from a test cycle I recently was on

NOTE – If you find an iPad bug: Please add [iPad – iOS xx] at the beginning of you bug title.

In this situation it is perfectly fine (and even required) that you include the environment in your title.

However, you may see something like this in the instructions:

BUG TEMPLATE: Please include the following info in all your bugs: Mobile device model and OS version Description of bug Wi-Fi or 3G / 4G?

This does not mean that all this information should be in the title. It simply means that it should be specified in the body of the bug. Generally you should put this information in the ‘Specified Environments’ or ‘Additional Environment info’ fields. It is the “Bug” template, not the “Bug Title” template.

Environment specific bugs are allowed
Occasionally a cycle will allow the same bug to be reported for different environments. In this case, each one of these bugs is considered different by the customer. Since the only difference between the bugs is the environment, it is necessary to include the environment in the title. Otherwise you would have multiple bugs with the exact same title and your fellow testers would have to look at the contents of the bug to see which environments had already been reported.

Keep Consistent with Earlier Bugs

Sometimes a cycle will ask you to include some extra piece of information in the title. One example of this would be the build of the application that you tested. What I usually see happen in these situations is every tester comes up with their own way of including this information. The result is a messy bug list that looks something like this:

[b 123] Area – Description
build 123 => Area – Description
Area – Description {build 123 v.2.045.34}
123 Area – Description

This is difficult for the customer and TTL to read and makes it impossible for them to quickly scan through the list.

Assuming that the earlier bugs followed the uTest standard and everything we addressed above, you should follow the pattern established in the first few bugs. Don’t worry about being original or sticking to your own personal preference, the goal is consistency. This will make the customer’s and TTL’s jobs much easier. See how much easier this is to read?

[b 123] Area – Description
[b 123] Area – Description
[b 123] Area – Description
[b 123] Area – Description

Learn From Other Testers

You can learn quite a lot from reviewing the bug reports of your fellow testers. You can see different styles of reporting the reproduction steps, come up with new ideas of how to test, and see which types of devices are the most common.

The same can be said for the bug’s title. When you are reviewing bugs, don’t just skip over the title. Instead, take advantage of the opportunity to learn from the mistakes and successes of others.

First evaluate the title of the bug first on its own:
Does the title follow the standard? Does it include appropriate key words?
Then look at it in the context of the entire report:
Does the title accurately and efficiently summarize the bug? Does it “sell” the importance of the bug?

As you pay more attention to your own bug titles as well as the titles of other bugs, you will start to see the types of patterns we have just talked about. It will become apparent that the testers who do these types of things are the ones that are separated from the crowd. Bug titles are extremely important and should be treated that way. Keep these tips in mind and you will be one step closer to writing the perfect bug report.

I’d love to hear your thoughts on this topic. What other tips can you give your fellow testers?

Reporting High-Value Bugs – Part 2

I originally posted on the uTest Forum.

In Part 1 of this series, we talked about the reasons why a uTester should focus on reporting high-value bugs. That led to some fantastic discussion and a spin-off thread about reporting every bug you find. Before you continue, you should go back and review those threads to get caught up on the topic.

In this Part 2, we are going to look at “HOW” you can find and report high-value bugs. This is a popular topic at uTest and there are many threads, webinars, and crash courses available (There are links to some of that material below). This post is intended to complement those resources and help us continue to improve our testing skills.

I’ve teamed up with fellow TTL and uMentor, Allyson Burk for a double dose of testing goodness :) We have some great ideas for you, so let’s get started!

Finding high-value Bugs

Focus on One cycle at a time (Allyson Burk)

I find there are two approaches to the workload at uTest: 1) accept every cycle, file a few bugs on each or 2) accept fewer cycles, file more bugs per cycle. Personally, I find the latter to be the best way to make more money, have more satisfaction in my work and increase my tester rating. Why? Because I can increase the quality of my work using this approach.

Giving myself more time on a product allows me to be methodical. I might use a few different approaches depending on the type of product.

– Deep, power user scenarios. I develop a goal in mind. A recent cycle I was on had a great example of this – you are a soccer mom and you need to equip your child for the upcoming season. This is going to yield the issues that are going to affect the target audience of the client. This approach can definitely yield high value bugs because you will be able to tell the client what is going to drive those target customers away.

– Break down the app into areas and dig deep. This is the approach I use when it is a newer, more unfamiliar application. I might spend a few hours in settings making sure each setting combination is functioning properly; or trying a variety of shopping cart, wishlist, checkout scenarios; or product customization. The key is not just spot checking to see if the area is functioning, but to really stretch the code and make sure all variables have been covered.

– Going down the rabbit hole. This is a less precise, more intuitive path where I just start investigating the parts of the application that I find interesting and following them as far as I can take them. If I really love the app or find it to be fun to use, this is the approach I will take. You have to be careful with this approach because you can “waste” a lot of time.

The key to all of these approaches is TIME. You cannot test in this deep manner if you do not have time and you cannot have time if you have 5-15 active cycles clamoring for your attention.

(Note from Lucas)
When you accept a new cycle, you are expected to thoroughly read the scope and instructions, read through the known bug list, review any other attached documents, and catch up on any chat posts. Then you have to set up your testing environment. You have to install the app, create an account, configure your proxy, etc. These start-up activities can be quite time consuming. Keeping your active cycles low allows you to spend less time getting ready to test, and more time testing.

Know the status of a project (Allyson Burk)

In general, clients are going to value bugs differently depending on the point in the development cycle they are on. It is important to pay attention to clues about where the client is in development when searching for high value bugs. This can be a moving target depending on the methodology used, agile vs. waterfall for example, but I think for this conversation we can think in terms of early, middle and late in the development cycle.

– Early in the development cycle, you can imagine that content related bugs are not going to carry huge value. The look and feel may still be in development, the final copy is likely not completed and images may not have been delivered. The client is rather going to be more focused on core functionality. They need to make sure the major functionality is there and working properly.

– Midway through the development cycle, functionality is still going to be the focus, but content starts to be more important. If ever there was a time to value spelling/grammar bugs, this would be it. Most copy has to get locked down for legal/translation/marketing/etc. so the client may be looking to make sure this is completely clean before shipping it off for various approvals.

– Late in the development cycle, stability and polish are key. Everything needs to be functioning at this time and the application needs to have a minimum of crashing/blocking issues. Many times in this last stretch before release of a product, the client might only be interested in High or Critical issues. The code will be fairly locked down at this point. The client will often not want to risk fixes that might break other functionality, so they are really going to be interested only in bugs that are of such severity to make the app unusable.

As uTesters, I think the trickiest aspect of this is knowing what phase of the development cycle the client is on. Logic might dictate that if you are on the first cycle for a new client that they would be early in the development cycle. I’d actually venture a guess and say that is actually almost never the case given my experience. I’d say we are usually brought in after the code is pretty stable and the content is beginning to be finished… somewhere in the mid stages.

But how can we know with more certainty?

Sometimes, this is as easy as reading the overview and paying attention to context clues. The PM might explicitly state, this is the first testable build of this product (early) or this is the release candidate (late). There may be things excluded from the scope like images (early to mid). There may be a very long known issues list (mid to late) or no known issues at all (early or late – HA this is a tricky one! They may clear all known issues for the later builds in order to make sure there has been no code regression before shipping the product out).

In the end, we will have to rely on the information provided and forge ahead. There is also never any harm, if you feel that there is no clear focus provided, to ask the question: Is there anything in particular the client is wanting us to focus on at this time? You might be refreshed at what avenues of testing that will open up for you.

Writing High-Value Bug Reports

Report bugs, not symptoms (Lucas Dargis)

The other day I was the TTL of a cycle and one of the features in scope was an account creation screen. The user was required to enter several pieces of information including their Address. Two different testers report these two bugs:

Bug 1 – Address field allows “!!!!!!!!!!!!!!!!!”
Bug 2 – Address field allows “!@#$%^&*()_+”

I see this type of thing all the time, so I know some of you saying “What’s wrong with that?”. The problem is both of these testers reported different symptoms of the same bug. If they had taken some time to do further investigation into the Address field, they would have realized that they hadn’t found a specific input that made it past the validation. They would have learned that the real issue was the Address field wasn’t being validated at all. The user could have entered anything (or nothing) and the system would have accepted it.

Whenever I encounter a bug, I spend a significant amount of time testing all around it, trying different inputs and different sequences of events until I  understand the root cause and all of its symptoms. This is where testers can show their worth. It’s easy to click on something and then report on the results, but it takes a much stronger skill set to be able to investigate potential bugs and then provide a valuable report of your findings. Customers can see this effort and they usually reward it.

Sell Your Bugs’s Prominence (Lucas Dargis)

If a bug is easy to find, it is usually more valuable then if it was an edge-case bug and it was unlikely that anyone would find it. Identifying your bug and reproduction steps is just the first step. The best testers know that how their bug report is written can affect how the customer views it’s prominence (how easy it is to find). The best testers keep their bug reports focused and their steps limited to the critical path. That means you should only list the specific actions needed to trigger the bug.

There is a problem with this approach. Often, bugs are hidden deep within the application and you might feel that you need to explain how you arrived at the bug. The way I get around this concern is to list “Prerequisite” steps at the top of the “Actions Performed” where I describe the starting state of the application.

Example:

Bug Title: Shopping Cart – Items added to the cart are not saved
Steps:
1. Go to the URL
2. Click on create new account
3. Enter a valid username
4. Enter and password
5. Click “Submit”
6. Log into the system with your account
7. Search for an item
8. Select the item
9. Add the item to my cart
10. View your shopping cart

The above report lists the steps from beginning to end, but it is fairly long and gives the impression that a user would have to do a series of very specific steps in order to find the bug. Instead, you should only list the steps that are directly related to the bug. Let’s see what that would look like.

Bug Title: Shopping Cart – Items added to the cart are not saved
Steps:
Starting state – User is logged into the application and viewing the details page for a product

1. Add the item to my cart
2. View the shopping cart

Explaining the starting state at the top of the report allows us to remove 8 steps. Now, because only the steps that specifically cause the bug are listed, this bug seems much more prominent and the report does a better job of highlighting the value of the bug. This is an oversimplified example but I hope you understand the point.

This is just one tip on how to sell your bug. This technique is called “Bug Advocacy” and is something ever tester should learn. To learn more about Bug Advocacy, here is a fantastic paper written by Cem Kaner:http://www.kaner.com/pdfs/bugadvoc.pdf

I want to thank Allyson for her contributions to this article. Please feel free to post questions, comments or challenges to anything we’ve written. Hopefully these ideas will prove useful to you in your quest for those high-value bugs.

Additional Resources

Be Creative: Bug-Hunting Tips from a Gold uTester (By Amit Kulkarni) – http://help.utest.com/testers/crash-cou … ld-uTester

How To Write the Perfect (uTest) Bug Report (by Rebecca Showerman and Nikki Sedgwick)- http://blog.utest.com/how-to-write-the- … t/2012/06/

How to Write a Good Bug Report (By Sunil Sidhwani) – http://forums.utest.com/viewtopic.php?f=55&t=3095

When a Bug is Not a Bug – Bugs vs Feedback (By Aaron Weintrob) – http://forums.utest.com/viewtopic.php?f=55&t=3179

Bug Reporting 101 (By Joseph Ours) – http://help.utest.com/testers/crash-cou … orting-101