2014 Year in Review

I’d love to write a witty post recapping the last year, but right now I just don’t have the time. So a brief summary will have to do for now.

Job Change

In February, I decided to leave Brock Solutions and work for uTest as a full-time consultant. I spent the next 8 months working for SalonCentric (a L’Oréal devision) as a business/test manager as they built and launched their new eCommerce website. That was wonderful time. I make some fantastic friends and learned a lot about testing eCommerce, particularly on the Demandware platform.

I was promoted to an Enterprise Test Team Lead which is essentially a contracted Test Project Manager. I was assigned several customers and project all of which were delivered successfully and received praise and accolades.


University – I wrote or co-wrote several courses for the uTest University

Forums – I stayed quite active on the forums as usual. I’m almost at 2,000 posts.

  • I started a really fun series titled Analyze This where I posted a testing “truth” and we debated it. Although technically I started this in 2013, the majority of the posts happened in 2014. So far I’ve started 13 topics, all of which earned at least 2,000 views and several are approaching 10,000!
  • I also made a very controversial proposal to change the uTest rating system to eliminate rating hits. There’s a lot at stake so I understand uTest’s hesitance to move on it, but I still think this is the right thing to do.

Blog – I wrote a few posts for the uTest blog, two of which were selected for The Best of 2014: Top Posts from the uTest Blog. “Certified and Proud?” was one of uTest’s most popular posts of the year, earning the largest number of comments and social shares. Even a few A-list testers read and talked about it!

A.C.E – uTest started a new initiative to provide testers with organized, mentor-based training. I helped put together the pilot program and was one of the mentors for it. A large portion of the materials we used were things that I’ve written for uTest over the years. I’m now working on converting all of my previous articles and papers into more structured courses as the A.C.E. program continues to evolve and improve.

Awards – uTest created a Hall of Fame website and introduced quarterly awards based on votes from the community. In the two quarters that this as been going on, I’ve won 3 awards

  • Outstanding Forums Contributors
  • Outstanding Blogger
  • Outstanding uTest University Instructor


In September, I successfully completed the Foundations Course. That was quite an experience. I hope to find some time to write about it one day.

In November, I passed the Instructors course and will be an assistant instructor for the first Foundations class of 2015.

uTest’s Testers of the Quarter (2014 Q4)

badgeTesterOfQuarteruTest recently announced the Testers of the Quarter winners for 2014 Q4.

This quarter I took home the hardware for:
Outstanding University Instructor because of the Exploratory Testing course I co-authored with Allyson Burk.

uTest University Webinar – Live Exploratory Testing

To go along with my course on Exploratory Testing, I had the opportunity to do a live webinar demonstrating how I do exploratory testing. The community management team did some great marketing and over 300 people signed up! Unfortunately, we found out at the start of the webinar that the uTest GoToMeeting license only supported 100 participants so a lot of folks were not able to attend. It was quite a humbling experience to have so many people interested in watching how I test.

As you can see from the video, the app I chose was quite buggy. That coupled with my poor internet connection presented some interesting challenges. It was difficult to maintain my train of thought and to investigate a single issue when other issues kept popping up, but I think it gave a good representation of what exploratory testing can be like sometimes.

Looking back at the video I need to improve in the following areas:

  • Speak slower and clearer – I was trying to articulate my every thought and this lead me to speak in quick, incomplete sentences. It might have been beneficial in showing how choppy my thought process can be, but I think it might also made it difficult to follow what I was thinking.
  • Repeat the question – During the Q&A portion, I didn’t do a good job of stating the question I was answering. I read the question and kinda mumbled it before jumping into the answer. I did OK on some questions, but I need to remember to do a better job of this next time.


uTest University Course – What is Exploratory Testing?

I recently authored a course with my good friend Allyson Burk for the uTest University on Exploratory testing. This is a heavily viewed course because it’s part of the “Getting Started at uTest” course track which most new uTesters work through. Since this course is a foundational piece of new uTester’s development, I spent a large amount of time researching and editing to make this as accurate and consumable as possible. So far it’s been well-received earning a 5-star average rating – Not too shabby.


It even got front-page billing on the University home page:
University Home Page - ET


Below is the content of the course:


Today’s development world is much different than it was 10 or even 5 years ago. Product development is fast-paced and must meet the high expectations of users. As development practices have changed, so too have approaches to testing. Many testers are finding that Exploratory Testing (ET) is an effective way to test in these circumstances. The adoption and use of ET has rapidly grown, to the point where it is arguably the most popular testing approach used today.

In this course we’ll start by learning what ET is and how it differs from scripted testing (ST). Then we’ll look at why you should use ET and finally, we’ll wrap up by showing you how to get started. So let’s make like Magellan and start exploring!


Before we look at ET, it might help if we first talk about a different, more traditional approach to testing so we can use that as a reference point to make some comparisons.

Scripted Testing (ST) is a two-step approach to testing. First the tests are written; they are planned, designed and documented. Second, the tests are executed. These two activities are done independently of each other and in many cases, the person who writes the tests is different than the person who executes them.

Generally, the tester executing the tests has some knowledge of the product, or the tests include the information needed to execute them. This is important because without that knowledge or information, the tester might not be able to execute the test or interpret its results.


Now lets compare that with ET which is simultaneous learning, test design and test execution. In other words, the tester is designing his or her tests and executing them at the same time. As an exploratory tester, your next action (the next test) is influenced by your previous actions, your observations of the product’s behavior, and your own thought process.

ET also assumes that a significant portion of the testing will be spent learning about the product. As you explore, you become more aware of how the product functions and its expected behavior. You can use that knowledge to design new and better tests. It also helps improve the analysis of the test’s results.

ET table

It is important to make the distinction between ET and other types of unscripted testing because some testers mistakenly believe that all unscripted testing is simply poking the product randomly to see what happens. Performing a series of random actions is called monkey testing and in some cases it may be a valid approach, however this is quite different from ET. With ET actions are the opposite of random — they are deliberate, driven by human thought and reasoning. Your approach is continually refined as new information is gathered and analyzed.

When an explorer goes to an uncharted region of the world, they spend months preparing. They go with a goal in mind and they rely on their abilities to adapt to changing situations. Similarly, an exploratory tester must prepare. They too have a goal and the skills needed to adjust their course. It’s true that monkey testing may occasionally find useful information, but it’s found unexpectedly. It’s the difference between discovery and exploration; luck versus skill.


One of the most defining qualities of humans is our ability to think. It’s what makes us unique. It allows us to analyze a situation and make decisions, or come up with new ideas, or find new solutions to a problem. We have the capability to learn and continuously improve.

Henry Ford once said “If you always do what you’ve always done, you’ll always get what you’ve always got.” Executing scripted tests over and over will generally produce the same results. Exploratory testers use their cognitive abilities to continually improve the value of their work. They explore and adapt, they learn and adjust. ET is designed to make the most of our intellectual abilities.

ET also takes advantage of the differences between testers. Each tester’s previous experience, skills, and thought process (among other things) causes him or her to view thing in a unique way. Different testers may come up with different ways to test the same function. It may be more beneficial to have three testers test the same function in three unique ways than it is to have all three test it in exactly same way.

In many cases we need to use ET because there is no other alternative. For example, consider a situation where the tester isn’t familiar with the product under test and scripted tests are not available. In this case, it’s up to the tester to study the product, design and execute their tests. As we’ve already seen, this is the very definition of ET.

It can take a significant time to read, comprehend, and execute each step in a test case. This is especially true if you don’t already know the product, or the TC uses language or terms you’re not familiar with. When a tester is tasked to find bugs quickly, they need to be searching for bugs, not reading test cases. They need the freedom to follow promising leads, not the constraints of predefined instructions.


If there is one thing all new testers (including new exploratory testers) should do, it’s to start by thinking about the product in general terms; try to see the big picture. Instead of initially focusing on one specific thing, first try to understand the context in which you are working.

Some questions to consider are:

  • Is this a product in development or is it already in production?
  • What is the purpose of the product?
  • Who are the users and how are they going to use it?

Jumping right in and banging on things might produce a bug or too, but if you hope to get the most out of ET, initial preparation and understanding your context is vital.

Now lets see how ET might look in practice. Imagine you’re a brand new tester, your boss comes to you and on your first day and says, “Here you go, this is the latest version of our app. Please begin testing and report any bugs you find.” There are no test cases and no documentation. What do you do? An exploratory tester would do something like this:

1. Get a notebook (or a digital word processor) to take notes as you go.

2. Explore the app as if you just downloaded it and want to use it yourself. If it is not an app you would typically use, then imagine you are the target market for the app.

Take a moment to really get in the mindset of a typical user. Some questions you can ask yourself are what is the goal of this app? Who would benefit from that? How do they benefit?

Let’s say this is an app to show up-to-date stock market information.

Goal of the app: Having stock market data at your fingertips.

Who benefits: Someone who is financially savvy or wants to be and has available income to be investing or has interest in other people’s investments.

How do they benefit: They benefit by the data being timely, accurate, easily accessible and displayed in a way that they can understand quickly.

Don’t worry about finding any bugs right now. You may stumble on them, but this is really just getting used to the app. Jot down anything you find that you want to explore further later.

3. Once you get a feel for the app start going back to the areas that interested you and you thought might be a place of vulnerability in the app. This knowledge about vulnerability is going to come with experience. Don’t worry if you don’t have any experience yet because you are about to get some!

4. One by one, work through each area you’ve earlier identified, exploring every function in that area. Think of what a real user might do. Come up with with some use cases or scenarios and execute those. Then think of variations and execute those. Use the results of your tests to help you come up with new ideas.

5. Focus on one bug at a time, but always be on the lookout for hints of other bugs or suspicious areas. In your notebook, quickly make a note of these areas and how to get back to them. This way you can come back and explore each one later. You could very well end up with 4 or 5 bugs just from investigating the initial bug.

6. Once you’ve exhausted that area or function of the app, move on to your next point of interest. As you repeat this process, remember what you’ve learned so far and use that information to influence your tests.

As you can see from this narrative, you are simultaneously learning, designing tests, and executing the tests. These are the core pieces of ET. Understand this and you’re on your way to becoming an exploratory tester.


Different testing needs call for different testing approaches and there are many situations where ET can prove most beneficial. ET is the inverse of scripted testing because it relies on human intellect as opposed to simply following instructions. ET is a process of continual refinement and improvement where testers adapt to situations and the information they’ve gathered. Now that you’ve been introduced to ET, our hope is that will continue to explore exploratory testing and that you can use these skills to provide the most value possible.


Webinar – Should testers report every bug they find?

In December of 2012, Ryan Lamontagne and I got into a good discussion on the uTest forms about whether testers should report every bug they find. We decided to kick it up a notch and debate it live in a webinar!


Webinar – Three more uTest Panel webinars

It’s been a busy past few weeks. In addition to picking up two new enterprise customer accounts (uTest TTL work) I was a panelist for three more uTest webinars.

Maximizing Your Benefit From The uTest Forums

Maximizing Exploratory Testing Methods

How to be a Quiet Tester That Customers Shout About


Checking vs. Testing – The Problem with Requirement Documents

The prevalent idea that testers are dependent on a requirements document to do their job is a dangerous one. Requirements are not always needed to test. In fact, in many situations, they may actually reduce a tester’s effectiveness.

The process of deriving tests directly from the requirements has several names. The ISTQB uses the term “specification-based testing”, sometimes it’s referred to as “Happy Path” testing, but I think the most appropriate name is “checking”. Michael Bolton wrote a well-known post about this topic (http://www.developsense.com/blog/2009/08/testing-vs-checking/). Checking is confirming that what we believe is actually true. Products are built in accordance with the requirements, so the requirements are what we believe to be true. When we verify that our product meets the requirements, we are “checking” the product. When a tester relies on a requirement document to test, he isn’t testing, he’s checking.

When we test, we are exploring, investigating and learning. Our actions are influenced by new questions and ideas that haven’t yet been explored. The use of requirement documents while testing can cause problems because it can give a false sense of test completeness, it can steer testers in the wrong direction, and it can reduce the independent thinking of the tester.

Gives a false sense of test completeness

If we have verified that our product meets all of the requirements, does that mean that the product has been well tested? True, you have verified that the product behaves the way we expect it to (in specific situations) but you still don’t know how the product will behave in situations not specified in the requirements.

“That wasn’t in the requirements” I’ve heard testers use this excuse many times and it drives me crazy. As a tester it is your job to investigate the product to learn about areas and behaviors outside of the requirements. A product is much more than its conformance to the requirements. It’s up to you to cover that gap between what is expected of a product and what actually is.

Checking that a product meets the requirements is necessary of course, but checking alone does not indicate test completeness.

Steers testers in the wrong direction

When you first start testing a new product or feature, what should be the first thing you test? Some might say you should verify the requirements. I challenge that view. I think the requirements should be one of the last things you test. In my opinion, a developer who writes code that didn’t meet the requirements failed to do his job. Before the test team sees the product, the developer has already spent hours working and verifying that his work is correct. Although possible, a strong developer rarely produces work that doesn’t meet the requirements. With that in mind, there is little value in retesting his work, especially when you consider the other aspects of the product that haven’t been tested yet.

Requirement documents can stifle the creativity of exploratory testing. When testers have a requirements document in front of them, they may be more likely to verify the requirements first and focus on areas where the likelihood of learning new information is the lowest. Instead, they should focus their efforts on new tests and unexplored areas where the opportunity for learning is the highest. When testing without requirements, you eliminate its influence on your testing decisions; you have to rely on your own abilities, your knowledge, and your curiosity. You test.

All testers have heard the old saying “Trust but verify”. I’m not saying that we shouldn’t verify developer’s work, only that there’s more value in performing a test for the first time, then performing the same test multiple times. Focus on the activities that produce the most value first.

Reduces independence

The idea of “independence” (See ISTQB Foundation) refers to how close a person is to a product. A developer who wrote the code would have the least amount of independence, while a person in a different company who has never seen the product would have the most independence. Independence can often be a great quality for testers. Testers with no preconceived notions of how the product is supposed to work are able to view the product with more objectivity.

Consider the common situation where a product was built according to incorrect requirements. The tester was able to verify that the product met the requirements, so was there an issue? In this case the requirements served as a false crutch to the tester. Just because the product met the requirements doesn’t mean it was working correctly. A tester with no knowledge of the requirements would have been better positioned to identify any errors because he wouldn’t have been comparing it to an incorrect “truth”.

We have now seen some reasons why testers should avoid relying on requirement documents. While they may be a necessity for different “checking” activities, testers who wish to provide value thorough “testing” must understand that their value is best realized when they test without requirements.