Month: April 2012

Quality is Customer Value – My Quest for the uTest MVT Award

One thing I respect about the uTest management is their continual pursuit of ways increase customer value. It’s an essential business objective to ensure the health and growth of our company. ‘Value’ should be the middle name of any good tester. “Lucas Value Dargis”. Sounds pretty cool huh?

I had just finished my 26th uTest test cycle. I had put an extra amount of focus and effort into this cycle because there was something special at stake. Occasional uTest offers a MVT award which is given to the most valuable tester of the cycle. The selection process takes several things into account including the quality of the bugs found, clear documentation, participation, and of course, customer value.

The MVT award not only offered a nice monetary prize, but it’s also a way to establish yourself as a top tester within the uTest community. I decided I was going to win that MVT award.

As usual, I started by defining my test strategy. I took the selection criteria and the projects scope and instructions into account and came out with these 4 strategic objectives:

  • Focus on the customer-defined ‘focus’ area
  • Report only high-value bugs
  • Report more bugs then anyone else
  • Write detailed, easy to understand bug reports
  • Be active on the project’s discussion board

When the test cycle was over I reflected on how well I’d done. I reported 9 bugs, more then anyone else in the cycle. Of those, 8 were bugs in the customer’s ‘focus’ area. The same 8 were also rated as very or extremely valuable. All the bugs were documented beautifully and I was an active participant in the discussion board.

There was no competition. No one other tester was even close. I had that MVP award in the bag. I was thinking of all the baseball cards I could buy with the extra Cheddar I’d won. I even called my mom to tell her how awesome her son was! You’ll can only imagine my surprise when the announcement was made that someone else had won the MVT award. Clearly there was some mistake right? That’s not how you spell my name!

I emailed the project manager asking for an explanation for this miscarriage of justice. The tester who won had fewer bugs, none of them were from the ‘focus’ area and they weren’t documented particularly well. How could that possibly be worth the MVT award? The PM tactfully explained that while I had done well in the cycle, the tester who won had found the 2 most valuable bugs and the customer deemed them worth the MVT award.

I was reminded that my adopted definition of quality is “Value to someone who matters” and suddenly it all fell into place. It didn’t matter how valuable I thought my bugs and reports were. It didn’t matter how much thought and effort I put into my strategy and work. At the end of the day a testers goal, his or her mission, should be to provide “someone who maters“ with the most value possible. I’m not that “someone who matters”. That “someone” is our customer.

It was a hard pill to swallow, but that lesson had a strong impact on me and it will be something I’ll carry with me moving forward. Congratulations to the MVT, I hope you enjoy all those baseball cards.

uTest Review From a New uTester

Back in March of 2012, I was listening to a podcast from SoftwareTestPro.com and they were talking about this thing called uTest. I decided to check it out.

What is uTest?

My first impression was “oh great, this is just another rent-a-tester website”, but as I looked a little closer I realized that this this company was much more powerful then I thought. uTest takes advantage of crowdsourcing (or in-the-wild testing as they call it) by distributing the work across people all over the world. In a matter of hours a website or app can be tested by dozens of different device configurations by real world users as well as professional testers.

Companies who have something they need tested hire uTest and their army of testers to perform the testing tasks. Testers are paid per bug found and per test case wrote or executed. The more valuable the customer finds the bug or test case, the more money the tester is paid.

This provides tremendous value for both companies and testers:

Companies get their products tested much quicker and cheaper then they could with in-house testers. They can have 100 testers working for one week vs. having 5 tester test for 20 weeks. Plus, they only pay for work they find valuable.

Testers get the opportunity to work on many different types of testing tasks, projects, environments and basically get exposed to things not available in their daily job. Getting paid is nice too.

Obviously there are downsides to this testing approach. For example, uTest testers don’t know or have time to learn your business or all of the intricacies of your product, so uTest shouldn’t be used when a deep level of understanding is needed. However, there are many projects where just having a hundred different eyes on on the product is exactly what is needed.

My First Test Cycle

The first project (called a test cycle) I worked on was an unpaid “interview” sandbox test cycle. This is an opportunity to try out to become a uTest tester. The task was to test NBA.com. There really wasn’t much in the way of requirements or instruction, except that you could only report 4 bugs and they had to be within the NBA.com domain. We would be evaluated on the value of our bugs and our ability to follow directions.

Since this was an audition, and I was limited to only 4 bugs I knew that each bug was very important for my evaluation. It’s hard to judge someone’s ability based on just 4 bugs so I needed to make their job easier. uTest rates bugs on a value scale: somewhat, very, and exceptionally. I wanted to be the top ranked tester in the cycle so in order to do that I thought about the assignment, the constraints, and the customer and came up with a plan.

My plan was to find different types of bugs in different locations within the site that would show my technical knowledge and testing expertise. Solid communication skills are a must in an online-based work environment so I’d need to make sure my bug reports where clear and descriptive.

Through my testing, I found dozens of bugs and I could have easily just logged the first 4 I found and been done with it. But I was very careful to make sure that I only reported those that fit into my strategy and represented my abilities as accurately as possible.

My value as a tester can be found in my strategy; how I went about finding and reporting bugs to accomplish my goal. All testing should be driven by a mission or it serves no purpose. In this case my mission was to be the highest rated tester so all my testing was done with that in mind. The actual work I did was really just “operational effectiveness” and should be expected for any good tester.

Results

  • I received the highest rating out of all 107 testers in the cycle.
  • I was the only tester who had all 4 bugs bug reports rated higher then ‘somewhat valuable’ ( I had the lone ‘exceptionally valuable’ bug report)
  • My overall uTest rank was higher then 33% of the 55,000 uTesters (Not too bad for only having 4 bug reports submitted)
Overall, I’d say my strategy worked out nicely. I don’t point this out to brag (well, not entirely), but to show the importance of a mission and a formulating a strategy to accomplish it.

Lessons Learned

From a customer’s point of view the power of crowdsourcing is amazing. The shear volume of bugs we uncovered in just a few days shocked me, especially considering how heavily trafficked that site is.

For testers, the opportunity to see the work of 100 other testers is priceless. There were several very solid testers in the cycle who had fantastic bug reports. I was able to glean some useful knowledge and techniques from reading through their reports.

The tester talent pool is sadly shallow. It surprised me just how clueless the majority of the testers were. Of the 219 bugs that were reported, 71 were rejected because the tester didn’t follow the simple instructions. It was obvious that the majority were clicking around aimlessly and reporting anything they could find.

The gap between a good tester and a bad tester is wider then I thought. Thankfully uTest takes this into account. Only testers ranked in the top 30% of the sandbox test cycle are allowed to join a paid test cycle. This allows uTest to filter out the majority of the poor testers. Also, lower ranked paid testers are limited in the amount and complexity of projects they can participate in while highly ranked testers are offered many more test cycles. This helps ensure projects have the best possible testers, keeps poor testers out, and gives new testers the opportunity to prove themselves.

I’m a now a raving fan of this company and crowd sourcing (or team sourcing as uTest calls it)!

Learn more about uTest at utest.com.
You can also check out my uTest profile here.