uTest University Course – What is Exploratory Testing?

I recently authored a course with my good friend Allyson Burk for the uTest University on Exploratory testing. This is a heavily viewed course because it’s part of the “Getting Started at uTest” course track which most new uTesters work through. Since this course is a foundational piece of new uTester’s development, I spent a large amount of time researching and editing to make this as accurate and consumable as possible. So far it’s been well-received earning a 5-star average rating – Not too shabby.

https://university.utest.com/what-is-exploratory-testing/

It even got front-page billing on the University home page:
University Home Page - ET

 

Below is the content of the course:

INTRODUCTION

Today’s development world is much different than it was 10 or even 5 years ago. Product development is fast-paced and must meet the high expectations of users. As development practices have changed, so too have approaches to testing. Many testers are finding that Exploratory Testing (ET) is an effective way to test in these circumstances. The adoption and use of ET has rapidly grown, to the point where it is arguably the most popular testing approach used today.

In this course we’ll start by learning what ET is and how it differs from scripted testing (ST). Then we’ll look at why you should use ET and finally, we’ll wrap up by showing you how to get started. So let’s make like Magellan and start exploring!

THE ‘TRADITIONAL’ APPROACH TO TESTING

Before we look at ET, it might help if we first talk about a different, more traditional approach to testing so we can use that as a reference point to make some comparisons.

Scripted Testing (ST) is a two-step approach to testing. First the tests are written; they are planned, designed and documented. Second, the tests are executed. These two activities are done independently of each other and in many cases, the person who writes the tests is different than the person who executes them.

Generally, the tester executing the tests has some knowledge of the product, or the tests include the information needed to execute them. This is important because without that knowledge or information, the tester might not be able to execute the test or interpret its results.

WHAT IS EXPLORATORY TESTING?

Now lets compare that with ET which is simultaneous learning, test design and test execution. In other words, the tester is designing his or her tests and executing them at the same time. As an exploratory tester, your next action (the next test) is influenced by your previous actions, your observations of the product’s behavior, and your own thought process.

ET also assumes that a significant portion of the testing will be spent learning about the product. As you explore, you become more aware of how the product functions and its expected behavior. You can use that knowledge to design new and better tests. It also helps improve the analysis of the test’s results.

ET table

It is important to make the distinction between ET and other types of unscripted testing because some testers mistakenly believe that all unscripted testing is simply poking the product randomly to see what happens. Performing a series of random actions is called monkey testing and in some cases it may be a valid approach, however this is quite different from ET. With ET actions are the opposite of random — they are deliberate, driven by human thought and reasoning. Your approach is continually refined as new information is gathered and analyzed.

When an explorer goes to an uncharted region of the world, they spend months preparing. They go with a goal in mind and they rely on their abilities to adapt to changing situations. Similarly, an exploratory tester must prepare. They too have a goal and the skills needed to adjust their course. It’s true that monkey testing may occasionally find useful information, but it’s found unexpectedly. It’s the difference between discovery and exploration; luck versus skill.

WHY DO EXPLORATORY TESTING?

One of the most defining qualities of humans is our ability to think. It’s what makes us unique. It allows us to analyze a situation and make decisions, or come up with new ideas, or find new solutions to a problem. We have the capability to learn and continuously improve.

Henry Ford once said “If you always do what you’ve always done, you’ll always get what you’ve always got.” Executing scripted tests over and over will generally produce the same results. Exploratory testers use their cognitive abilities to continually improve the value of their work. They explore and adapt, they learn and adjust. ET is designed to make the most of our intellectual abilities.

ET also takes advantage of the differences between testers. Each tester’s previous experience, skills, and thought process (among other things) causes him or her to view thing in a unique way. Different testers may come up with different ways to test the same function. It may be more beneficial to have three testers test the same function in three unique ways than it is to have all three test it in exactly same way.

In many cases we need to use ET because there is no other alternative. For example, consider a situation where the tester isn’t familiar with the product under test and scripted tests are not available. In this case, it’s up to the tester to study the product, design and execute their tests. As we’ve already seen, this is the very definition of ET.

It can take a significant time to read, comprehend, and execute each step in a test case. This is especially true if you don’t already know the product, or the TC uses language or terms you’re not familiar with. When a tester is tasked to find bugs quickly, they need to be searching for bugs, not reading test cases. They need the freedom to follow promising leads, not the constraints of predefined instructions.

HOW DO I GET STARTED?

If there is one thing all new testers (including new exploratory testers) should do, it’s to start by thinking about the product in general terms; try to see the big picture. Instead of initially focusing on one specific thing, first try to understand the context in which you are working.

Some questions to consider are:

  • Is this a product in development or is it already in production?
  • What is the purpose of the product?
  • Who are the users and how are they going to use it?

Jumping right in and banging on things might produce a bug or too, but if you hope to get the most out of ET, initial preparation and understanding your context is vital.

Now lets see how ET might look in practice. Imagine you’re a brand new tester, your boss comes to you and on your first day and says, “Here you go, this is the latest version of our app. Please begin testing and report any bugs you find.” There are no test cases and no documentation. What do you do? An exploratory tester would do something like this:

1. Get a notebook (or a digital word processor) to take notes as you go.

2. Explore the app as if you just downloaded it and want to use it yourself. If it is not an app you would typically use, then imagine you are the target market for the app.

Take a moment to really get in the mindset of a typical user. Some questions you can ask yourself are what is the goal of this app? Who would benefit from that? How do they benefit?

Let’s say this is an app to show up-to-date stock market information.

Goal of the app: Having stock market data at your fingertips.

Who benefits: Someone who is financially savvy or wants to be and has available income to be investing or has interest in other people’s investments.

How do they benefit: They benefit by the data being timely, accurate, easily accessible and displayed in a way that they can understand quickly.

Don’t worry about finding any bugs right now. You may stumble on them, but this is really just getting used to the app. Jot down anything you find that you want to explore further later.

3. Once you get a feel for the app start going back to the areas that interested you and you thought might be a place of vulnerability in the app. This knowledge about vulnerability is going to come with experience. Don’t worry if you don’t have any experience yet because you are about to get some!

4. One by one, work through each area you’ve earlier identified, exploring every function in that area. Think of what a real user might do. Come up with with some use cases or scenarios and execute those. Then think of variations and execute those. Use the results of your tests to help you come up with new ideas.

5. Focus on one bug at a time, but always be on the lookout for hints of other bugs or suspicious areas. In your notebook, quickly make a note of these areas and how to get back to them. This way you can come back and explore each one later. You could very well end up with 4 or 5 bugs just from investigating the initial bug.

6. Once you’ve exhausted that area or function of the app, move on to your next point of interest. As you repeat this process, remember what you’ve learned so far and use that information to influence your tests.

As you can see from this narrative, you are simultaneously learning, designing tests, and executing the tests. These are the core pieces of ET. Understand this and you’re on your way to becoming an exploratory tester.

CONCLUSION

Different testing needs call for different testing approaches and there are many situations where ET can prove most beneficial. ET is the inverse of scripted testing because it relies on human intellect as opposed to simply following instructions. ET is a process of continual refinement and improvement where testers adapt to situations and the information they’ve gathered. Now that you’ve been introduced to ET, our hope is that will continue to explore exploratory testing and that you can use these skills to provide the most value possible.

ADDITIONAL READING

Key Decision: Become a full-time uTester

Decision

So I’m about 7 months late in writing this, but the key decision I made back in January was to leave my job as the test lead for Brock Solutions and become a full-time uTester.

Working at Brock, I enjoyed the people I was working with and was reasonably content with the company. But he product was the most untestable and convoluted piece of software I’ve ever seen, which resulted in extremely challenging work. It was difficult to add value as most of the day was spent trying to get the system ready to test. The work wasn’t what I thought it was going to be when I signed up and at the end of the day I didn’t enjoy my job.

In early January, Steve Moses, a Sr. project manager at uTest contacted me. He said he had an exciting opportunity to be a dedicated test lead for one of uTest’s largest clients. It was a 6-month contract with the possibility of an extension. It really wasn’t much of a decision. How could I turn down representing my favorite company, the one I spend my free time working for anyway?

Expectations

  • Build my reputation as as a top uTester and leader in the community
  • Develop a better understanding of e-commerce
  • Gain experience working with people from industries
  • Influence the education and growth of others
  • Find more challenging and rewarding career options

How to Write a Quality Bug Report

I worked with Aaron Weintrob to put together a uTu course on how to write quality bug reports. The intended audience of this course is new uTesters with little or no testing experience. We kept it relatively short and simply highlighted the key areas. We may create additional courses for various sections where we can dig in a little deeper. Here is a link to the course. Below is the course’s content:

INTRODUCTION

Writing a good bug report is one of the most talked-about topics in the testing world. The art of creating a well-written bug report requires a balanced combination of testing and communication skills. This course provides advice and tips geared towards helping you create bug reports that are informative and actionable, thus improving their value to the customer.

WHAT MAKES A GOOD BUG REPORT?

Most testers understand the role of a bug report is to provide information, however a “good” or valuable bug report takes that a step further and provides useful information in an efficient way.

To help us get started writing valuable bug reports, we are going to focus on a few key areas:

  • The Title
  • Actions Performed (Steps)
  • Expected and Actual Results
  • Attachments

THE TITLE – THE GOOD, THE BAD, THE UGLY!

The title is the face of your report. It’s the first thing anyone sees and it’s importance cannot be overstated. A good title helps reduce duplicate issues and can quickly convey a summary of the bug.

It’s best to avoid generic problems in the title. For example, these should never be used:

  • XYX is not working properly
  • Issue with XYZ
  • XYZ is corrupted/does not look right

The above example titles add little value in describing the problem. By nature, every report is describing something that is not working as it should. Be specific about what makes it “not working.”

Instead of: Sorting is not working properly.
Try: Sorting is happening in reverse order.

Instead of: Issues with GUI on navigation bar.
Try: Navigation bar is wrapping to a second line.

Often times, bugs are migrated into the developer’s database that may contain hundreds, if not thousands, of other issues. Imagine trying to search this database for “navigation bar”. That search will return every issue related to the navigation bar. Searching for “wrapping to second line” is much more specific making it easier to find the bug. Your bug report needs to survive (and be useful) beyond the current test cycle; a strong title will help it through it’s journey.

ACTIONS PERFORMED – ADVICE FOR EXPLAINING YOUR STEPS

This is the body of your report. The goal of this section is to show the reader how to reproduce the bug. Since this area usually contains the most information, it’s important to keep it concise and easy to read. Always number your steps and kept them short and to the point.

Tip: Using a prerequisite can reduce the number of steps.
Instead of listing out every step to login in, start your steps with: “Prerequisite: User is logged in”

Tip: Find the direct path to the bug
Often times, testers will stop at the point where they found a bug and log their last few actions. However, the most helpful bug reports are those that distill the report down to the core reproduction steps.

It’s a good exercise to reproduce the bug by following the steps you’ve just outlined. This will help ensure you’ve included everything the customer will need to reproduce it as well.

Sometimes digging a little deeper below the surface of the bug can add additional value. Here are some examples of how adding a bit more effort or thought will produce a higher quality report.

Example 1: Provide additional useful information
Scenario: You find that a video does not play.
Good: Mention if it happened on all videos and not just the one mentioned in report.
Better: Specify if the issue is reproducible on more than one browser or device.
Best: Upload a speed test showing that bandwidth was adequate when testing was happening.
Lesson: Try to identify and answer follow-up questions before the customer asks them.

Example 2: Report the bug, not a symptom of the bug
Scenario: We are testing an Address input field. We find that the Address field allows “1234567890″ and it also allows “!@#$%^&*()_+”
Lesson: These are two different symptoms of the same bug. Closer inspection would reveal that the real issue is the Address field isn’t being validated at all. The problem may be more serious than the first symptom you find.

EXPECTED AND ACTUAL RESULTS – WOULDA, COULDA, SHOULDA

Now that you have described how to reproduce the bug, you need to explain the problem and the desired behavior.

Tip: When describing expected results, explain what should happen, not what shouldn’t happen.
Instead of: The app shouldn’t crash.
Try: The user is taken to XYZ screen.

Tip: When describing actual results, describe what did happen, not what didn’t happen.
Instead of: The user wasn’t taken to the XYZ screen.
Try: The user remained on the ABC screen.

ATTACHMENTS – WHAT TO DO AND WHAT NOT TO DO

Attachments add to the bug’s value by offering proof of the bug’s existence, enabling the customer to reproduce it or helping the developer fix it. Each attachment should add to the value of the bug in at least one of these three ways.

The following are some tips and guidelines to keep in mind when adding attachments:

IMAGES

  • Adding images is a quick way to add context to your bug. Consider adding an image even if you also have a video.
  • Highlight the area(s) of interest in your image.
  • Attach the image files directly to the report. Don’t put images in a Word document or a zip file.
  • Use images to illustrate static issues.

VIDEOS

  • Video confirms your steps were accurate at the time the issue was created. For example, a screen grab of an error message isn’t as useful as seeing what went into creating that error message.
  • Actions in the video should match the steps listed in the bug report.
  • Videos should be trimmed to only show the bug.
  • Provide video if the steps are complex.
  • External/live videos can be more impactful than mirrored videos because you can see hand gestures or you touching a button on the screen.

LOG FILES AND OTHER TIPS

Avoid proprietary file types (like .docx). Use .txt instead.
Avoid compressed (.zip) files unless specifically asked for or approved by the TTL, PM, or customer.

UTEST ETIQUETTE – GENERAL OVERVIEW OF PROPER BEHAVIOR

It is important to remember that you are representing the TTL, the test team, and all of uTest when you work on the test cycle. Your fellow testers rely on you to write a good title for your bug report so they won’t file a duplicate bug. TTLs depend on clean, good reports to ensure the customer receives value from the cycle. uTest needs quality work from everyone so we can continue to work in the field we all love.

ADDITIONAL READING

Here are some valuable discussions about bug reports from the uTest Forums:

Two contributions to the uTest University

Back in December of 2013, uTest officially launched the uTest University (blog post) which is intended to be a single source for testers of all experience levels to access free training resources. This is a neat opportunity for testers to contribute to the growth and development of the testing community by creating courses and writing articles. The university also offers  Author Page

My first course was derived from a uTest forum’s post I I wrote back in June of 2013. I was on a cycle where the customer required logs from Charles Web Debugging Proxy be attached to every bug report, but none of the testers (myself included) or the knew what that was or how to use it. I spent some time learning how to use the tool and then put together a tutorial to share with the rest of the team. Fast forward 8 months later several other customers started required the same thing. To make the information a bit easier to find the tutorial was turned into a uTu (uTest university) course:
How to Set Up Charles Web Debugging Proxy for iOS Devices and Windows 8

My second course came at the request of the uTest Community Management team. They needed a tutorial for new testers to show them how to create videos (screencasts) of their bugs. They specifically wanted it based around the free tool Screencast-O-Matic. I had actually never used that tool before, so I spent some time getting familiar with the tool. I also compiled a list of suggestions and tips based on things I see frequently in the videos of other testers. The result is:
How to Set Up and Use Screencast-O-Matic

 

A Tester’s Haiku

From time to time uTest puts on fun contests for the community. This month the objective was to come up with the best testing-related haiku. Over 90 testers participated, coming up with some great poems ranging from serious to silly to clever.

There was one haiku that got more attention than most so I felt it warranted some further discussion. It was one of the first submissions and it was well-written so the uTest CM team used it to promote the contest. It was mentioned it on the forums and posted it on Twitter. It was nominated as one of the 10 finalists and went on to win the contest, earning the vote of 21 other testers. Here is the poem in question:

Gifted are testers
Fixing a world of errors
Coded by others

It follows haiku syllable structure and has a nice flow, but I feel it’s misleading and potentially damaging. I’ll quickly address each line.

Gifted are testers
Can’t argue with that :)

Fixing a world of errors
This shouldn’t be a  great revelation, but in general, testers do not fix errors; We provide information and dispel misconceptions. One of the way’s we do this is by identifying errors. Some might try to make an argument that part of resolving a problem is finding it, but testers who understand what we do to provide value, realize the importance of making the distinction between finding an fixing a bug. These are two different actions that required two different skillsets.

Coded by others
This line really bothers me because it promotes an us vs. them mentality. Even though someone else (the developer) did write the code, it doesn’t help to point the finger at them. It’s a testers job to identify mistakes in other people’s work and that puts us in a delicate situation. It needs to be handled without judgment or condemnation. Bugs are an inevitable part of software development and since quality is the responsibility of the entire team, we need to work together, take responsibility together and avoid blaming each other.

Now, I’m not trying to say the author of this haiku meant to say that testers literally write the code to fix bugs or that developers and testers are enemies and I don’t mean to pick on him/her, but the fact that so many other testers thought this haiku was a good representation of testing troubled me. I’m sure this can be partially attributed to uTest’s endorsement and also because of language issues, but these are two themes that I see often and wanted to briefly address them.

My Entry
In case you were wondering, I did write a haiku. While I was a finalist, I didn’t make it into the top 3. Here’s my entry:

found and reported
a customer enlightened
value provided

Webinar – Should testers report every bug they find?

In December of 2012, Ryan Lamontagne and I got into a good discussion on the uTest forms about whether testers should report every bug they find. We decided to kick it up a notch and debate it live in a webinar!

http://forums.utest.com/viewtopic.php?f=13&t=4430

Testing with the Microsoft Surface RT

About the Device

The Microsoft Surface RT is a interesting device to test with. It has the new Windows 8 interface, but it also still has the traditional desktop interface because not all things can be accomplished on the Windows 8 side. This can make things a little confusing, especially if you are just starting to test with this device. So, here are some instructions to help you get ready to set up and back to testing in no time.

Installing Apps

When we are testing apps that are in development, they are not available in the Windows Store so we will need to manually install them. Usually, you will be given a zip file that contains all the files you need.

  1. Download the zip file
  2. Go to your Desktop and locate the zip file (Should be in your Downloads folder)
  3. You can’t install directly from a zip file so you need to extract the files
  4. Long tap/release (or right-click if you have a mouse) to open the context menu
  5. Tap on ‘Extract All…”
  6. Select the location where you want the files saved to and tap ‘Extract’
  7. The extracted folder will open
  8. Locate the Windows Powershell Script (should have a Notepad icon)
  9. Long tap/release (or right-click if you have a mouse) to open the context menu
  10. Tap ‘Run with PowerShell’
  11. You’ll see a series of prompts, accept all of them
  12. If this is the first time you’ve installed an app, you’ll be required to sign up for a development license. It’s free and it has just a few simple steps
  13. After the application is installed, the PowerShell windows will just close. There is no indication that it was installed, but when you go back to your home screen, you should see the app on the far right

Uninstalling apps

  1. On the home screen, drag the app icon down just a bit until a check mark shows up just above the icon
  2. Release the app and the action menu will open from the bottom of the screen
  3. Tap ‘Uninstall’ to remove the app

Error logs

The Surface RT essentially is a PC, so application errors and crashes will be logged in the Event Viewer just as they are on a regular Windows desktop or lap top.

How you access the Event Viewer on the Surface RT is more complicated than on a regular PC since there is no Start button or search function. Here is how you can add a link to the Event Viewer to your desktop:

  1. From the Home screen, tap on ‘Desktop’
  2. Long tap/release (or right-click if you have a mouse) to open the context menu
  3. Select Personalize
  4. In the ‘Search Control Panel’, search for Event Viewer
  5. Tap on Administrative Tools(or Navigate to Control Panel > System And Security > Administrative Tools)
  6. Long tap/release (or right-click if you have a mouse) on Event Viewer to open the context menu
  7. Tap Send To > Desktop (create Shortcut)

Now that you have the event viewer open, you can view error messages and save the error files to attach to your bug reports.

  1. On the left pane, tap on Windows Logs >> Application
  2. On the right pane, tap on ‘Filter Current Log…’
  3. Select ‘Error’ and tap ‘OK’
  4. Select the error log(s) you want and on the right pane, tap ‘Save Selected Events’
  5. Save the events wherever you like and you can attach them to your bug reports

Taking Screenshots

Screenshots are pretty easy to take. You need to press the home button (Windows icon on the front of the Surface) and the down volume button (left side of the Surface) at the same time.
The file will be stored in your Photos > Pictures library > Screenshots folder

Taking Video

A well-created external video is usually the best way to go as I’ve done in all the above videos. Check out this post for some tips to improve your external videos.
I haven’t found a way to create mirrored videos yet, but I’ll update this if I do.

 

Webinar – Finding bugs in mobile devices

I was able to join Kayla Cox and Todd Smith for a uTest webinar to talk about testing mobile devices and how to find high-value bugs.

Since my microphone was terrible (and I might have been mumbling a little ) here is a summary of the points I made in our discussion.

Crashes
Understand that not all crashes are valuable.
Out of memory crash may be due to other apps using up 90% of your memory and the app you are testing just pushed you over the limit. The best way to know for sure, is to have a clean test bed. Restart your phone after you install a new app, and make sure no other apps are running in the background.

When you do get a memory related crash, use a memory management app to help you see where your memory usage spikes. Being able to identify a reproducible memory crash is usually a high-value bug

Connection Issues

  • Kill your connection while data is being transferred
  • Unplug your wi-fi router/modem
  • Turn on airplane mode
  • Turn of wi-fi on your device
  • Turn off cellular data on your device
  • Find places near you that have low or no signal and test there

Interaction with native and popular apps

  • Share something via email with no email set up
  • Log in using Facebook account with/without the Facebook app installed
  • Interrupt testing with phone calls, text messages, FaceTime calls etc
  • If the app changes the phone settings, make sure it does it correctly. Change it back manually in settings and see how the app responds

Investigation and Documentation
There are many topics on how to write good bug reports but there are a few points worth reiterating

  • Provide exact reproduction steps
  • Do root cause analysis – don’t report symptoms. I once saw 3 testers reported 3 different symptoms of the same bug. On the surface they all looked like different bugs, but a little analysis showed they were all caused by the same step they all overlooked. 

Some Thoughts on Functional Testing

I wrote this as a Guest Blogger for the uTest blog.

There is an age-old expression that says “You only have one chance to make a first impression.” This is a hard truth in today’s world of instant gratification. If your product fails to deliver the first time, your customers will simply move on to the next thing. In-the-wild functional testing, as provided at uTest, is similar to a dress rehearsal for your application. Your application is exposed to a group of people who accurately represent your potential user base. They can identify and report the issues (that would have negatively impacted your customer’s first impression) before your customer ever has the chance to see them.

A functional tester has the ability to evaluate individual features of an application. They are familiar with typical application behavior and have the skills needed  to look objectively at a feature and see what’s wrong.

Perhaps even more valuable is a functional tester who is able to analyze individual pieces of an application within the context of the entire application. A functional tester looks at a particular item, identifies integration points between that item and other parts of the application, and then formulates a plan of how to inspect those touch points. Applications are usually weakest in places where different parts come together. A strong functional tester knows this and knows how to exploit those weaknesses to identify any lurking bugs.

Functional testing will only be successful if an organization’s underlying quality fundamentals are solid and everyone clearly understands how testing helps achieve the goals of the business. Functional testing is only one of many activities that collectively comprise a comprehensive testing strategy. Depending on the needs and expectations of your company, different testing activities such as performance, load, and security testing should be considered. Functional testing differs from other types of testing in that it most closely reflects the experience of the users. While performance affects the experience and security issues add risk to the experience – how the application functions IS the experience.

I’m uTest’s 2012 Mentor of the Year!

For the past 3 years, uTest recognizes uTesters who have consistently gone above and beyond their call of duty. uTest recently announced their selections for the 2012 testers of the year and I was selected as the 2012 Mentor of the Year!

Wow! What a thrill!

As I’ve mentioned many times, uTest provides us testers with many opportunities to grow and develop our testing skills. We are constantly exposed to new products, devices, and customers. The uTest forum always keeps us up to date on the latest testing trends and hot debate topics. But uTest offers us more than opportunities to learn; uTest also provides a platform for us to teach and mentor.

My greatest thrill comes when uTesters comment on how one of my posts helped or inspired them. It’s the motivation behind everything I write. It’s a privilege to be able to influence new uTesers as they evolve into highly-skilled and respected testers.

uTest has assembled a community of testers ready to learn, but that need must be met by those willing to teach. Every tester has knowledge they’ve gained through study and experience. No matter how simple it may seem, that information is valuable. If you’re brave enough to share what you have learned, you’ll experience the amazing feeling of knowing you are positively impacting your community and industry.

I am truly honored to receive this award and I want to extend my sincere thanks to the uTest team and the uTester community.

If you care to read any of my “uMentor” posts, they are all located here.