Tag: review

Is uTest a Scam?

There are some reviews out there from testers claiming uTest is a scam and that you can’t make any money. I’ve also seen a few uTest customers complain about the quality of the testing and uTest’s sales/negotiating practices. These concerns are valid and I can understand why some people have the impression that uTest is not what it claims to be. In order to address these perceptions, we need to look at them from two points of view; The view of the customer, and the view of the tester.

The uTest customer

Complaint #1: The quality of the testing was lower than expected

I have no problem saying that there are a lot of bad testers at uTest. It’s true, there is no point in denying it. These testers only report low-value bugs, they don’t follow instructions, and generally disrupt the test cycle. Sometimes it is because they’re inexperienced testers, sometimes they’re just plain bad. Unfortunately this is one of the downsides of crowdsourcing. To uTest’s credit, they do realize this and are continuously working to improve the skills and abilities of uTesters. They also identify and remove problem testers.

On the other hand there are some absolutely awesome testers at uTest. These top testers consistently provide the customer with excellent service and high-value bugs. uTest does a pretty good job of identifying the strong testers and ensuring that they are the ones working on your projects. Keep in mind that there are literally hundreds of projects, dozens of Project Managers and thousands of testers from all over the world, so every test cycle is going to be different.

So what’s a customer to do? First you need to manage your expectations. Understand the limitations and benefits of uTest and make sure they align with your testing needs. Second, you need to be involved. Yes, uTest is a service, but the quality and success of your project is a direct result of your participation and influence.

Here is an excellent article from Elena Houser on how customers can get the most from their uTest (or any crowdsourced) testing service. It is a MUST read for any potential uTest customer: http://trancecyberiantester.blogspot.com/2012/10/crowdsourced-testing-lessons-learned.html

Complaint #2: The uTest sales process is shady

In full disclosure, I’m not a uTest customer so I’ve never gone through this process myself. However, I am a uTest TTL (Team Test Lead) and so I have worked with many different customers. I’ve seen customers who are extremely satisfied and those constantly complain. It doesn’t take long before you start to see a pattern.

I really could just do a copy/paste from above. Again, this comes down to having correct expectations and being involved. Customers who follow Elena’s advice will find that uTest’s services are well worth the money and effort. Those who don’t will be disappointed with their results.

Another point worth mentioning is that uTest is a start-up company. They have only been around a few years yet they are growing and changing incredibly fast. In just the last year I’ve seen some impressive improvements. I have no doubt that as the company matures and customer needs and expectations are better understood, the sales experience will improve and mature as well.

The uTest tester

Complaint #1: uTest is a scam

As I mentioned above, uTest is a start-up. The company grew faster than many people expected and as a result it went through some obvious growing pains. Everything about the company was (and still is) evolving. The payment process was still being worked out, bug reporting and evaluation was confusing, and in general the tester experience gave some the impression that uTest was either a scam or just unprofessional.

Admittedly, the first tester interface was terrible. It was slow, buggy, and difficult to use, which is quite ironic for a software testing company. This problem has now been addressed. uTest recently launched their new tester platform and it is so much better (read more here). Their are now several reliable ways for testers to receive payment, and there is an entire team of employees solely dedicated to the welfare of the testers. These are just a few examples of how uTest is working to improve its image and show testers that uTest is a legitimate company and a great place to work.

Complaint #2: You can’t make any money

I recently read a review from a uTester that he had reported 87 bugs but he only was paid for 16 of them. The other 71 bugs were rejected. He felt that bugs were intentionally rejected in order to avoid paying the testers. He’s not the only one to complain that testers are not adequately compensated for their efforts. Fortunately it is because of a few misconceptions.

Testers need to understand the uTest bug payment model. Customers pay uTest a set price for an agreed upon amount of work. It is up to the customer to accept or reject the bugs reported by the testers. uTest then pays the testers for the bugs (and other work) the customer accepted. Since the customer pays a flat fee no matter how many bugs are reported or accepted, they have no financial incentive to reject individual bugs. Bugs are rejected for valid reasons, not to avoid paying for them.

The other important point is testers are not paid for their efforts (there are some exceptions), they are paid primarily for the value they provide. Testers who provide the customer with high-value bugs make a lot of money. Testers who report low-value or “junk” bugs make very little money.

The bottom line is good testers can make good money working at uTest. Poor testers will be frustrated.

Conclusion

uTest is not a scam. It is a legitimate company and an amazing one at that. While uTest is not perfect, most criticisms can be answered if you look at the entire situation objectively.

2012 Year In Review

2012 was a career year for me. For the past 8 years I’ve just had a job. I didn’t really enjoy what I was doing and didn’t put much thought into how I could or should develop my career. Things changed quickly early in the year as several opportunities came together. Here are a few of those highlights:

New Jobs
I found the best job I’ve ever had, working as the principle tester at a semiconductor manufacturer. Before I arrived, there was no formal testing in the IT department. I was tasked with introducing testing in one group and then over time, grow it throughout the organization. I’ve been able to test the new “Flagship” application which is a few weeks away from our first Release. So far we have received rave reviews on all aspects of the application and the development process.

I was able to expand my testing skills by learning the nuances of SPA (single page application) testing. This has been a fun and challenging experience, mostly because it isn’t done much yet so there few resources out there geared specifically toward SPA testing.

I was also able to dabble in automation testing for SPAs. Since this type of application is client-side heavy, the true value comes from exercising it through the browser. Many automation solutions and supporters prefer testing the code directly (via APIs or a test harnesses), bypassing the browser. That has made this learning process more of a struggle for me then I had expected.

If you follow my blog at all you’ll know I’m also a uTest fan boy and freelance tester. I’ve already wrote a blog post about my uTest experiences this year, so I’ll just give an updated summary:

  • Team Test Lead
  • Became a solid iOS tester
  • Worked with and learned from testers all over the world
  • 200 cycles/425 bugs
  • 94% bug approval rate/44% high-value rate
  • Gold Rated (99.75%)

Improved Online Presence
One of the most valuable and educational aspects of this year was my decision to join and contribute to the testing community. I started this blog to chronicle my career development. I only found the time for 11 posts but I was able to post regularly… well kind of.

I spent most of my time focused on the uTest community. I became a uMentor, a forum moderator and one of the most active forum members. My topics have generated hundreds of responses and over 20,000 views. I’m now seeing more and more new uTesters step up and contribute to the growth of the forum and the uTest community which is fantastic!

I was also featured on the uTest blog a few times:

I have learned so much from writing about testing, teaching new testers, and learning from others. I’d say that focusing on developing my online presence has had the largest impact in my growth as a professional tester.

Scrum Mastery
In addition to testing, I’m also extremely interested in software development processes and improving efficiency, specifically the Scrum development framework. Since I had a few years of Scrum experience, I volunteered to be a Scrum Master for the “flagship” product I mentioned above. As word of that project’s success got around, I became a champion for Scrum in our organization. I was able to coach POs, Developers, Customers, and Managers and am currently Scrum Mastering 2 projects. I was asked to give an “Introduction to Scrum” presentation to our department during one of our Lunch & Learn sessions and since then one group has started their own project using Scrum.

To complement and improve my real-world experiences, I attended a Scrum Master course and then passed both the Scrum Alliance and Scrum.org Scrum Master certifications.

So cheers 2012; you’ve been swell. I look forward to meeting you 2013. I know you have many fun and challenging experiences in store.

STPCon Fall 2012 – My Review

I recently was able to attend STPCon in Miami. It was my first conference and I really enjoyed it. Here are some of my highlights.

People I met

I finally got to meet a few members of the uTest staff. Jessica, Matt, Mike, and Chris were all there slinging swag and preaching the uTest gospel. It was nice to get some face time with those folks. Unfortunately we didn’t have a uTest meet up. Maybe next time.

I did get to meet a former uTester, Joseph Ours. I talked with him for a while about his experiences with uTest and how he has made the transition into management and consulting. Joe is a knowledgeable and well-spoken guy. I really enjoyed getting to know him. Too bad he’s not an active uTester anymore, I would love the chance to work with him.

Hands-on vs. Lecture format

There was a track of 7 classes that focused on hands-on practicals. It seemed like a cool idea to me, so I spent the first 3 sessions in hands-on classes. While they were interesting and it was fun to test with other testers, I didn’t really learn anything. It was more like “Here’s a program, lets think of ways to test it”. I do that every day when I test with uTest. I wasn’t there to test, I was there to learn how to become a better tester.

While I can see the value of the hands-on classes for people who don’t have the luxury of contently testing new things with new people, for me it wasn’t the best use of my limited and expensive time. (I was there on my own dime)

After I realized that, I switched over to the lecture/discussion sessions and really found a lot of value there. Most of those sessions weren’t lectures but more of someone sharing their experience or suggesting ideas on how to do things better. The dialog between the presenters and the audience was also great. There was lots of idea sharing and discussions.

Favorite Sessions

Mike Lyles of Lowes gave a two part talk about how Lowes created a Test Center of Excellence (TCoE). It was extremely useful for me to see what a mature testing organization looks like and how it functions. It helped me start to develop my long term vision for my testing organization.

I also enjoyed Joseph Ours’s talk about “Redefining the purpose of software testing”. Simply put, he was making the argument that testing teams should be looked at as information providers, not gate keepers or decision makers. I have been making this argument for a while now but had been looking at it the wrong way. I thought it just was the definition of what testers should do. His explanation showed me that it actually can be an effective way to explain the value that testing provides. Testing is a service. We provide information to make informed decisions.

One lady in the audience challenged Joe saying that she is the test lead as well as the gate keeper and that in her organization it works fine. Joe and her debated a bit but she wasn’t convinced. After the presentation Joe and I chatted about that point more and realized that we need to think of roles and people separately. A testers role is to discover and provide information. A decision makers role is to make decisions based, in part, on that information. Usually those role are divided between two different people (or groups of people) but in some cases, it may be the same person.

In her case, she led the testing team, but she also had additional business knowledge and the authority necessary to be able to decide when the product was fit for delivery.

Not so great

The Vendor showcase was disappointing. There were only 10 or so booths. I talked to most of them in under an hour and spent the rest of the time trolling around the uTest booth sharing my uTest testimony with anyone who would listen 🙂

Overall

I’m glad I was able to make it down to STPCon. I got to meet some great people, learn a few things, and get a better understanding of our industry in general. It was a valuable experience and am looking forward to attending next year. Maybe I’ll see you there.

uTest – My First 100 Cycles

So I’m a little late, I’m actually at 138 cycles, but I wanted to give an update on my uTest experience now that I’ve got 100 cycles under my belt.

Accomplishments

When I first signed up with uTest I set a few goals. I really had no idea how realistic they were, but you have to at least have something to shoot for right?

By the end of 2012 (9 months from when I started) I wanted to:

  1. Earn my gold badge in Functional testing
  2. Become a TTL (Test Team Lead)
  3. Develop a strong reputation within the uTest community

Gold badge

I got my functional badge within 30 days of my first test cycle. At first this was actually a disappointment. I was really looking forward to the challenge of having to work hard for that badge.

After the initial shock wore off I decided that it actually was an accomplishment to be proud of. I realized that uTest has created an efficient system that allows strong testers to bubble up to the top quickly which makes sense when you think about it. Why would they want to hold good testers back? You want your best testers out front setting an example and representing the company.

TTL

Three months in, I got the email I had been anxiously waiting for; I had been invited to be a TTL! A rite of passage to become a TTL is you have to manage a sandbox class. Working that cycle was the most difficult thing I’ve done at uTest so far. Working with and evaluating 100 rookie testers, triaging their 400+ bugs and test cases in just a week is grueling to say the least. But the sense of accomplishment at the end made it all worth it. Since then, it’s been a fantastic experience working closely with the PMs while helping guide the testers so they can develop and become successful.

Reputation

My approach to this goal was to focus on the uTest form. The forum is an amazing community full of talented testers from all over the world. Every day there is are several interesting conversations going on. We constantly learn new things while challenging and encouraging each other.

After a few months of active participation, I was asked to become a uMentor and a forum moderator. Basically my role is to spark discussions and debates as well as write educational “mentoring” posts.

Before I started working at uTest maybe 10 other people knew I was a tester (That includes my mother). Now literally hundreds of people read my posts and engage in awesome discussions with me every week. So far, the conversations I’ve started have generated over 13,000 views.

I’m not sure I’ve fully achieved this goal yet, but I’m on my way.

Testing Statistics

To quantify my time at uTest so far and to brag a little 🙂 here are my digits:

  • Products Tested: 70
  • Test Cycles: 138
  • Bugs Filed: 342
  • Bug Acceptance: 93.2%
  • High Value Bugs: 47%
  • Functional Rating: 99.3% – 99.7%

You can see my uTest profile here: https://my.utest.com/platform/profile/LucasDargis

uTest Perception

Without a doubt, joining uTest has been the biggest factor in the development of my career. I have grown more in the last 7 months then I ever expected. I was able to achieve all my goals and then some.

Here is a short list of the amazing opportunities uTest provides testers:

  • The opportunity to work for every type of company. You are able to see how testing is done from start-ups to big Fortune 100 companies. Every cycle adds to your experience and helps open your mind to the many different ways testing is approached.
  • The opportunity to work with and learn from people from all over the world. I’ve made friends in the UK, India, Romania, Brazil, etc. The global exposure is priceless.
  • The uTest forum is relatively new and the number of active contributors is still small, but since the uTest community is so large (60,000+ testers) the potential is huge.  There is an excellent opportunity for testers to get in on the ground floor and quickly establish themselves in a community that is evolving into an industry leader.
  • There are several leadership opportunities. You can become a TTL, a forum moderator, a uMentor, and a Gold-rated tester. Being active in discussions on the form or in your test cycle (providing help, answering questions) are informal ways to develop your leadership skills.

You may have noticed that I haven’t talked about money at all. That is because for me, getting paid at uTest is an extra bonus. I can always make money, but it’s these other aspects that provide the most value to me as a tester.

100 cycles down. Next stop, 1000 bugs!

uTest Review From a New uTester

Back in March of 2012, I was listening to a podcast from SoftwareTestPro.com and they were talking about this thing called uTest. I decided to check it out.

What is uTest?

My first impression was “oh great, this is just another rent-a-tester website”, but as I looked a little closer I realized that this this company was much more powerful then I thought. uTest takes advantage of crowdsourcing (or in-the-wild testing as they call it) by distributing the work across people all over the world. In a matter of hours a website or app can be tested by dozens of different device configurations by real world users as well as professional testers.

Companies who have something they need tested hire uTest and their army of testers to perform the testing tasks. Testers are paid per bug found and per test case wrote or executed. The more valuable the customer finds the bug or test case, the more money the tester is paid.

This provides tremendous value for both companies and testers:

Companies get their products tested much quicker and cheaper then they could with in-house testers. They can have 100 testers working for one week vs. having 5 tester test for 20 weeks. Plus, they only pay for work they find valuable.

Testers get the opportunity to work on many different types of testing tasks, projects, environments and basically get exposed to things not available in their daily job. Getting paid is nice too.

Obviously there are downsides to this testing approach. For example, uTest testers don’t know or have time to learn your business or all of the intricacies of your product, so uTest shouldn’t be used when a deep level of understanding is needed. However, there are many projects where just having a hundred different eyes on on the product is exactly what is needed.

My First Test Cycle

The first project (called a test cycle) I worked on was an unpaid “interview” sandbox test cycle. This is an opportunity to try out to become a uTest tester. The task was to test NBA.com. There really wasn’t much in the way of requirements or instruction, except that you could only report 4 bugs and they had to be within the NBA.com domain. We would be evaluated on the value of our bugs and our ability to follow directions.

Since this was an audition, and I was limited to only 4 bugs I knew that each bug was very important for my evaluation. It’s hard to judge someone’s ability based on just 4 bugs so I needed to make their job easier. uTest rates bugs on a value scale: somewhat, very, and exceptionally. I wanted to be the top ranked tester in the cycle so in order to do that I thought about the assignment, the constraints, and the customer and came up with a plan.

My plan was to find different types of bugs in different locations within the site that would show my technical knowledge and testing expertise. Solid communication skills are a must in an online-based work environment so I’d need to make sure my bug reports where clear and descriptive.

Through my testing, I found dozens of bugs and I could have easily just logged the first 4 I found and been done with it. But I was very careful to make sure that I only reported those that fit into my strategy and represented my abilities as accurately as possible.

My value as a tester can be found in my strategy; how I went about finding and reporting bugs to accomplish my goal. All testing should be driven by a mission or it serves no purpose. In this case my mission was to be the highest rated tester so all my testing was done with that in mind. The actual work I did was really just “operational effectiveness” and should be expected for any good tester.

Results

  • I received the highest rating out of all 107 testers in the cycle.
  • I was the only tester who had all 4 bugs bug reports rated higher then ‘somewhat valuable’ ( I had the lone ‘exceptionally valuable’ bug report)
  • My overall uTest rank was higher then 33% of the 55,000 uTesters (Not too bad for only having 4 bug reports submitted)
Overall, I’d say my strategy worked out nicely. I don’t point this out to brag (well, not entirely), but to show the importance of a mission and a formulating a strategy to accomplish it.

Lessons Learned

From a customer’s point of view the power of crowdsourcing is amazing. The shear volume of bugs we uncovered in just a few days shocked me, especially considering how heavily trafficked that site is.

For testers, the opportunity to see the work of 100 other testers is priceless. There were several very solid testers in the cycle who had fantastic bug reports. I was able to glean some useful knowledge and techniques from reading through their reports.

The tester talent pool is sadly shallow. It surprised me just how clueless the majority of the testers were. Of the 219 bugs that were reported, 71 were rejected because the tester didn’t follow the simple instructions. It was obvious that the majority were clicking around aimlessly and reporting anything they could find.

The gap between a good tester and a bad tester is wider then I thought. Thankfully uTest takes this into account. Only testers ranked in the top 30% of the sandbox test cycle are allowed to join a paid test cycle. This allows uTest to filter out the majority of the poor testers. Also, lower ranked paid testers are limited in the amount and complexity of projects they can participate in while highly ranked testers are offered many more test cycles. This helps ensure projects have the best possible testers, keeps poor testers out, and gives new testers the opportunity to prove themselves.

I’m a now a raving fan of this company and crowd sourcing (or team sourcing as uTest calls it)!

Learn more about uTest at utest.com.
You can also check out my uTest profile here.