uTest Review From a New uTester

Back in March of 2012, I was listening to a podcast from SoftwareTestPro.com and they were talking about this thing called uTest. I decided to check it out.

What is uTest?

My first impression was “oh great, this is just another rent-a-tester website”, but as I looked a little closer I realized that this this company was much more powerful then I thought. uTest takes advantage of crowdsourcing (or in-the-wild testing as they call it) by distributing the work across people all over the world. In a matter of hours a website or app can be tested by dozens of different device configurations by real world users as well as professional testers.

Companies who have something they need tested hire uTest and their army of testers to perform the testing tasks. Testers are paid per bug found and per test case wrote or executed. The more valuable the customer finds the bug or test case, the more money the tester is paid.

This provides tremendous value for both companies and testers:

Companies get their products tested much quicker and cheaper then they could with in-house testers. They can have 100 testers working for one week vs. having 5 tester test for 20 weeks. Plus, they only pay for work they find valuable.

Testers get the opportunity to work on many different types of testing tasks, projects, environments and basically get exposed to things not available in their daily job. Getting paid is nice too.

Obviously there are downsides to this testing approach. For example, uTest testers don’t know or have time to learn your business or all of the intricacies of your product, so uTest shouldn’t be used when a deep level of understanding is needed. However, there are many projects where just having a hundred different eyes on on the product is exactly what is needed.

My First Test Cycle

The first project (called a test cycle) I worked on was an unpaid “interview” sandbox test cycle. This is an opportunity to try out to become a uTest tester. The task was to test NBA.com. There really wasn’t much in the way of requirements or instruction, except that you could only report 4 bugs and they had to be within the NBA.com domain. We would be evaluated on the value of our bugs and our ability to follow directions.

Since this was an audition, and I was limited to only 4 bugs I knew that each bug was very important for my evaluation. It’s hard to judge someone’s ability based on just 4 bugs so I needed to make their job easier. uTest rates bugs on a value scale: somewhat, very, and exceptionally. I wanted to be the top ranked tester in the cycle so in order to do that I thought about the assignment, the constraints, and the customer and came up with a plan.

My plan was to find different types of bugs in different locations within the site that would show my technical knowledge and testing expertise. Solid communication skills are a must in an online-based work environment so I’d need to make sure my bug reports where clear and descriptive.

Through my testing, I found dozens of bugs and I could have easily just logged the first 4 I found and been done with it. But I was very careful to make sure that I only reported those that fit into my strategy and represented my abilities as accurately as possible.

My value as a tester can be found in my strategy; how I went about finding and reporting bugs to accomplish my goal. All testing should be driven by a mission or it serves no purpose. In this case my mission was to be the highest rated tester so all my testing was done with that in mind. The actual work I did was really just “operational effectiveness” and should be expected for any good tester.

Results

  • I received the highest rating out of all 107 testers in the cycle.
  • I was the only tester who had all 4 bugs bug reports rated higher then ‘somewhat valuable’ ( I had the lone ‘exceptionally valuable’ bug report)
  • My overall uTest rank was higher then 33% of the 55,000 uTesters (Not too bad for only having 4 bug reports submitted)
Overall, I’d say my strategy worked out nicely. I don’t point this out to brag (well, not entirely), but to show the importance of a mission and a formulating a strategy to accomplish it.

Lessons Learned

From a customer’s point of view the power of crowdsourcing is amazing. The shear volume of bugs we uncovered in just a few days shocked me, especially considering how heavily trafficked that site is.

For testers, the opportunity to see the work of 100 other testers is priceless. There were several very solid testers in the cycle who had fantastic bug reports. I was able to glean some useful knowledge and techniques from reading through their reports.

The tester talent pool is sadly shallow. It surprised me just how clueless the majority of the testers were. Of the 219 bugs that were reported, 71 were rejected because the tester didn’t follow the simple instructions. It was obvious that the majority were clicking around aimlessly and reporting anything they could find.

The gap between a good tester and a bad tester is wider then I thought. Thankfully uTest takes this into account. Only testers ranked in the top 30% of the sandbox test cycle are allowed to join a paid test cycle. This allows uTest to filter out the majority of the poor testers. Also, lower ranked paid testers are limited in the amount and complexity of projects they can participate in while highly ranked testers are offered many more test cycles. This helps ensure projects have the best possible testers, keeps poor testers out, and gives new testers the opportunity to prove themselves.

I’m a now a raving fan of this company and crowd sourcing (or team sourcing as uTest calls it)!

Learn more about uTest at utest.com.
You can also check out my uTest profile here.

4 thoughts on “uTest Review From a New uTester”

  1. Hi Lucas,
    Thanks for the very interesting article :)
    I’ve been with uTest for a while now (I’m currently a Gold rated test team lead with 98.4% rating).
    I have to say though I disagree with your example of one of the down sides to uTest for two reasons: Firstly, many of the products we test are continuously tested over a number of cycles during their development and as testers we develop an intimate understanding of the project and the needs of the customer. Secondly, for more complex products uTest recruits teams of skilled testers from the pool (in some cases recruiting testers with specific specialist skills such as security testing etc.) and give them in depth instruction as required. I’ve personally been involved in a few of these cycles where an intricate understanding of the product is essential straight off the bat. From my perspective as a tester the true down side to the uTest approach is that in most cases you only get paid for the bugs you find and not the work you’ve put into finding them. So the rate of return for your effort decreases exponential with the number of cycles the product has been through. So the incentive to actually test a product drastically decreases as it matures unless there are test cases involved, which have guaranteed payouts for completion.
    I must admit I was not shocked at your assessment of the talent involved in the sandbox cycle. This is not a true image uTest’s tester pool though as the sandbox is intended as a rough way of assessing new recruits and since all you have to do to be invited to a sandbox cycle is sign up with uTest many of those involved will be clueless :) The true assessment of testers comes from their work in the field and this is reflected in their rating. So in essence all uTest testers are continuously assessed.
    Congrats on being the top tester in your sandbox. I totally agree with you that all testing should be done with some strategic goal in mind. I hope to see you in a test cycle 😉

  2. I have been using utest at my company for 6 months. The way they do business is completely shady and we wasted hours of our time dealing with contract issues. We paid for local testers and found out they were outsourcing instead. We are now canceling our contract as they are simply unreliable and we are so tired of all the back and forth negotiation. Save yourself time and go somewhere else.

    1. I’m not a uTest employee so I can’t speak to your contract problems, but I’d like to address you local tester complaint. uTest puts a high value on providing the right testers from the right locations. uTest requires testers verify their location and only offer them work if they are in the location specified by the customer. That said, testers occasionally falsely claim they live in the US to increase their chances of getting more work. This is a violation of uTest’s code of conduct policy and testers will be permanently banned from uTest if they do this. I suspect this is what happened in your case.

Leave a Reply

Your email address will not be published. Required fields are marked *