Tag: Testing

5 Traits of successful uTesters – Part 1

Originally posted at: https://www.utest.com/articles/5-traits-of-successful-utesters—part-1

The Ability to Cope with constant Change

There are many traits that testers must have in order to be successful. A quick Google search for “traits of a successful tester” will produce several articles on the subject. However in this article series, I wanted to focus on the traits that are of particular importance to uTesters.

At Applause (uTest) our approach to testing is quite unique. Testers who had success at a typical testing shop are not guaranteed to have success here. Conversely, people with no prior testing or even IT experience, often become highly respected and sought after uTesters… providing that have these particular qualities.

This will be a 5-part series and in this Part 1, we are going to discuss the importance of the ability to cope with constant change.

Probably one of the most unique aspects of being a uTester is that we test an incredibly large and ever-changing set of products. Instead of working for one company and testing the same product(s) over and over, uTesters test hundreds of different products for hundreds of customers every day. Some of the top uTesters have literally tested over a thousand different products.

The top uTesters work on several products at any given time. In order to be successful, they need to be constantly analyzing the context that they’re working in. Which product am I testing? What is the state of the product? What is the goal of this cycle? If a uTester fails to test within the correct context, they’ll find that their work provides little or no value.

Just as every product is different, so too are the needs and expectations of every customer. One customer may have extremely specific reporting standards. They require bug titles to follow a detailed format, they have attachment requirements and expect several pieces of additional data. However, the next customer may not have any requirements and is just happy to be receiving bug reports at all.

uTesters need to be able to quickly switch from context to context and from one set of customer expectations to another, often several times per day. This is no simple task, but it’s the first trait all the best uTesters share.

uTest TTL Handbook

The Fundamentals for All Test Team Leads

This handbook explains the basic concepts surrounding the uTest Test Team Lead role.

Developed by uTest Community Management  – 2/11/2014

Introduction

The term TTL is a uTest-derived acronym that stands for ‘Test Team Lead.’ TTLs are hand-selected members of the uTest community who work side-by-side with uTest Project Managers (PMs) to facilitate the flow and outcome of test cycles. A few traits and required skills of good TTLs include:

  • Hard skills: high-quality testing output with proven track record, attention to details, expertise managing test cycles based on customer requirements across multiple technologies, Experience with uTest and an understanding of test cycle operations, and device/industry specific expertise
  • Soft skills: excellent communication skills, professionalism, culturally sensitive, fair and helpful
  • Sense of urgency: attentive to test cycle details and activity, focuses on responsiveness, escalates issues accordingly to the PM or CM, constructive not confrontational
  • Business acumen: understands what’s important to the customer with the ability to analyze and communicate the information across the testing community when necessary

Although the tasks required per test cycle may differ, the goal is the same: to increase the value that uTest brings to the customer by maximizing the output and minimizing the noise level.

Distribution of TTL Work

A TTL is vetted and trained before being placed onto paid projects. Paid projects are assigned by a PM. TTL payouts are always defined upfront, before any work is performed, and are kept between the TTL and the PM. Each assignment is different and depends on many factors, such as the TTL’s ability to triage bug reports or their ability to create test cases, to name two. Other criteria may include the TTL’s experience in a particular testing domain, industry, or mobile platform and access to those platforms to reproduce bugs. TTLs should utilize the TTL Availability Google Doc to enter your weekly availability and project preferences. Request access through the link provided.

Details of TTL Assignments

For many test cycles, the TTL’s main responsibilities are to monitor tester questions via test cycle chat and triage reports as they come in. TTLs should be in regular contact with the PM to ensure that the test cycle is headed in the right direction and that the customer is delighted with the results. Below are details of the responsibilities TTLs may be asked to perform:

  1. Platform Communications with Testers and Customers
    No matter the type of communication, keep in mind that testers must follow the uTester Code of Conduct at all times. TTLs should monitor the chat as closely as possible in order to address incoming questions and deliver outgoing announcements, especially in the first few hours that the test cycle launches.

    • Chat: Test cycles typically have real-time chat enabled, which allows TTLs to communicate with testers, customers, and project managers in real time. There are three components of chat: Questions about Scope, Announcements and Threaded/Topic Conversations.
      • Questions on Scope – for general questions on the cycle. This is usually related to scope questions, but can contain other general questions about the cycle such as questions on difficulty installing the build or claiming a test case.
      • Announcements – where TTLs should place their welcome message to the testers (see example welcome message in the addendum). Also, this is where the TTL would make any other announcements such as: announcing a new build, announcing a new out of scope section, making a correction to the information in the overview, reminding testers about the cycle close time, encouraging testers to get test cases submitted, refocusing the team on the focus areas.
      • Topic Conversations – discussions isolated to a specific topic. If there is a major development that the TTL thinks should have more attention than just an announcement, consider starting a new thread. These Topics can garner more attention because the TTL can edit the title. Threaded conversations are also good for topics like test case claims. Using these topics can keep the Questions about Scope section a bit cleaner.
        NOTE: All chat activities can result in an email sent to the testers and the customer depending on their settings, so please be mindful of the noise that chat can create.
    • Tester Messenger: Notes that TTLs add to Tester Messenger will result in an email directly to the tester who reported the issue and is added to the defect report for easy reference. This message is visible to the customer, PM, and any other testers on the test cycle. TTLs should use the Tester Messenger when:
      • There is a question about the defect that requires an answer from the reporting tester.
      • The testers will sometimes leave a message here for the TTL or the customer when the TTL, PM or customer leaves them a message in the tester messenger. Be aware that testers cannot initiate that exchange. In addition, the TTL receives an email any time a message was added to a bug that they have commented on. When testers leave a message for the customer, the customer should have visibility and may respond, but if historically they are not that responsive, the TTL can email the Project Manager, relaying the question, then respond to the tester in the Tester Messenger.
        NOTE: Be careful about exposing the TM’s contact information to the tester if responding to the tester via email (say, forwarding the response). The TTL is never to give out the TM information unless expressly asked to do so by the PM. If so, please send the information to the tester privately through email or chat to keep that information out of the tester messenger where it could be seen by other testers. It is preferred for uTest to be the go-between rather than to start opening doors between the testers and TMs.
      • If the question and answer are determined to be valuable to other testers by the TTL, share this information on the Chat/Scope & Instructions with all the testers.
    • Request More Info: If there is missing information or more clarification is required from a tester, the TTL can push a bug report to the “Request More Info” status, which shows that the bug report was reviewed but requires a response from the tester. Be aware that CM recommends using this status only for small portions of missing information. For a bug that has more significant issues, for example it does not follow the customer’s bug template, is clearly a placeholder, or is just poorly written, the bug should be rejected as “Did Not Follow Instructions.” TTLs are encouraged to assist the overall community in elevating the expectations of testers and eliminating consistently poor bug documentation performance by rejecting bugs that meet those criteria.
      NOTE: Sending reports into the “Request More Info” status automatically adds a note to the Tester Messenger and prevents testers from discarding these bugs. However, it is possible to visually note when a tester has responded through the Platform UI and through email notification as the status of the bug returns to “New”. In addition, this status assists the testers when they see their bug change status. Testers should be reminded to press “Send Requested Info” button. Please note that if the tester does not respond before the cycle closes, then their bug will be automatically rejected.
    • Customer Notes: Notes that you add to Customer Notes sends an email to the customer’s Test Manager and makes a copy of the message in the defect report for easy reference. This message is visible to the customer, PM, and any TTLs on the test cycle who have TTL permissions (but not to testers). TTLs should use the Customer Messenger when:
      • The defect needs to be exported to the customer’s tracking system
      • The defect has been exported to the customer’s tracking system and the TTL needs to leave a note to identify the corresponding defect ID# in the other system.
        NOTE: The key to keeping track of what needs to be exported versus what has already been exported is to be consistent. For example, the TTL leaves a Customer Note only when the defect has been exported to the customer’s tracking system. Then in that note, simply enter the corresponding defect ID# from the other system. If all of the defects will not be exported to the customer’s tracking system, or if the defect is being approved but is not being exported to the customer’s tracking system, the TTL can insert a message to denote that.

        • The TTL needs to communicate directly with the customer about this issue, such as to provide clarifying information that you may have discovered after your initial recommendation was provided or to respond to a request or question from a customer, as examples.
      • Direct Email: uTest Project and Community Managers place a high level of trust in our TTL community. Occasionally TTLs will be given access to confidential and/or proprietary information including tester email address. TTLs must ensure they treat this confidential information with the upmost sensitivity.
        NOTE: If you are emailing more than one tester at a time you must place email addresses into the BCC field. This will ensure that tester email addresses remain hidden from fellow community members.
  2. Triaging Issues
    If possible, triage reports as they come in (or in short intervals of time) to prevent a pileup of work. Doing so allows the TTL to easily spot duplicates and potentially identify areas of the application that may need further clarification from the customer (in these cases, contact the PM immediately). Additionally, testers will appreciate timely triaging to help direct them where to focus their testing efforts for subsequent testing. There are two levels of permissions for TTLs: Basic and Approval. The Basic level allows the TTL to send an approval recommendation for how an issue should be triaged, while Approval allows the TTL to effectively triage as a customer or PM. Basic-level permissions require the TTL to complete several pre- determined dropdowns that provide more information to the customer and PM – including whether the issue is valid, in-scope, reproducible, and a duplicate of a previously-reported issue (hereby labeled the TTL Recommendations Workflow). Please consider the following items in triaging issues:

    • Triage fairly and consistently: The age-old question is whether to offer testers a 2nd chance when they do not follow instructions, or use a ‘tough love’ approach that will signify the importance of high quality with every single interaction. In an effort to save time and resources, our preference is to offer the ‘tough love’ approach, unless the report is salvageable and could be valuable for the customer. Therefore bugs that violate the customer specified bug templates should be rejected as “Did Not Follow Instructions”, while high-quality bugs and reports that have minor mistakes can be salvaged using “Request More Info” tools (more on that below). Testers can discard their bugs up until the TTL triages their bugs, including sending it to the “Request More Info” state.
    • Bug Templates – Is the report in the proper format (if the customer asked for a specific format)?
      • Confirm that the bug title contains the appropriate information.
      • Confirm that the steps to reproduce are clearly documented as required by the customer or development. Testers should be urged to number their steps to make it easier on the customer to read and reproduce.
      • Confirm Expected Results and Actual Results are documented clearly and provide factual information.
      • Confirm that all appropriate attachments are included as required.
      • Confirm that all the additional environment information is included.
      • Confirm that all other customer specific format requirements are met.
      • Confirm that the bug is classified appropriately.
    • Bug Reproducibility – Is there sufficient evidence supplied in the bug report?
      • i. When possible, TTLs should attempt to reproduce each submitted issue. In some cases the TTL is not able to try to reproduce a defect before triaging it due to device or time constraints. In these instances, the TTL should ensure the tester has provided sufficient proof that it happened during their testing (logs, screenshots, videos, clear and complete steps, etc.).
      • ii. If the TTL is unable to reproduce the issue on the same device/OS as the tester, they may consider asking the tester to verify the issue is still happening to verify the steps written are full and complete. They may have either missed a step or the issue may have corrected itself. Given the bug has adequate information to verify the tester was seeing the issue at the time it was reported, non-reproducibility should not be a reason to reject a tester’s issue before the customer has reviewed it.
      • iii. If there is missing information or more clarification is required from a tester, the TTL can push a bug report to the “Request More Info” status, which shows that the bug report was reviewed but requires a response from the tester.
        Note: Sending reports into the “Request More Info” status automatically adds a note to the Tester Messenger and prevents testers from discarding these bugs. However, it is possible to visually note when a tester has responded through the Platform UI and through email notification as the status of the bug returns to “New”. In addition, this status assists the testers when they see their bug change status. Testers should be reminded to press “Send Requested Info” button. Please note that if the tester does not respond before the cycle closes, then their bug will be automatically rejected.
    • d. Bug Classification – Is the severity and bug type of the issue appropriate?
      • i. If unsure, clarify with the PM about severity and value expectations. For example, what do they consider “Critical” vs. “Low”?
      • ii. If the Severity is not appropriate, send the tester a message asking them to update it, or the TTL can update it themselves if they have the appropriate permissions in the platform. TTLs can also leave a message for the customer’s Test Manager to make the update when they are approving bugs.
      • iii. If the type (functional, UI, technical) is not appropriate, send the tester a message asking them to update it, or the TTL can update it themselves if they have the appropriate permissions in the platform. TTLs can also leave a Customer Message for the customer’s Test Manager to make the update when they are approving bugs.
    • e. Duplicate Bugs – Is the issue a duplicate of a previously-submitted issue?
      • i. If you believe it is, reject the issue as a duplicate. Be sure to cite the bug report # that the rejected issue is a duplicate of. The time stamp for each defect entered into the platform is critical. When deciding which defect is a duplicate and which defect to approve, bugs reported earliest take precedence.
      • ii. If the customer does not want the TTL to do any actual approvals or rejections, the TTL can recommend that the defect be rejected as reason Duplicate and provide the # of the original defect for reference.
    • f. Out of Scope – Does the bug appear to be in scope?
      • i. The scope is determined by the customer and should be clearly defined in the Scope and Instructions provided by the PM.
      • ii. If the bug is not in scope, recommend rejection using “out of scope” as the reason.
      • iii. If the bug is not in scope and the TTL has the ability to approve and reject, then simply reject the bug using the “out of scope” reason.
    • g. Bug Validity – Does the bug appear to be valid?
      • i. Most submitted defect reports will be clearly valid or invalid for one of the reasons above, but some will need further consideration. It may be worthwhile to try to reproduce anything you are in doubt of, to get a clearer idea of what issue is actually being reported.
    • h. Bug Approvals – if the TTL has the ability to approve bugs, they should be very careful that they approve the bug at the proper value. Do not accidentally approve a bug as Exceptional when it should be Very or Somewhat valuable.
    • i. Customer Approvals – How to deal with the reported issue that is a valid defect from a tester standpoint but not a valid issue from the customer’s perspective?
      • i. In this scenario, there are two actions that the TTL can take, depending on the individual circumstances:
        • i) Reject (or recommend rejection) as Working as Intended/Designed. This reason does not negatively impact the tester’s rating but they also will not get paid for the defect.
        • ii) If applicable, Approve (or recommend approval of) the defect but leave a Customer Note that indicates that it should not be imported into the customer’s tracking system. Defect value is normally set to “Somewhat valuable” for these.
      • ii. In the case where the test application is either overly complicated or the customer does not have a complete list of known issues/working as designed, it’s more fair to the testers to approve the defect so they are paid for the time they put in reporting it. Too many “Working as designed” rejections cause testers to shy away from testing and can cause participation rates to drop.
      • iii. If the average tester would consider the issue as working as designed, then a rejection is the appropriate strategy. Use your best judgment when making this decision.
    • j. Bug Export – If the bug is valid, does it need to be entered into the customer’s bug tracking system?
      • i. If so, and the TTL will be handling the bug tracking system import, there are two ways this can be done.
        • i) If the account is set up for it, you can use the “Export to Tracking” button on the bug report in the platform to automatically move the report into the customer’s system. This functionality is customer facing. If a TTL is given an account to access customer site they must be aware that they should not use that account to triage bugs. TTLs should only use their tester/TTL account to triage issues. Under no circumstances should TTLs approve their own bugs.
        • ii) If the account is not set up for it, you can manually copy the bug report from the uTest JIRA system to the customer’s bug tracking system. The most efficient way to do this is to click the Copy link at the top of the defect screen in the platform and paste it into WordPad, then copy the sections you need into the corresponding sections in the customer’s tracking tool (JIRA/Team Track/etc.).
  3. Triaging Test Cases
    Depending on the test cycle and PM, TTLs may be responsible for approving or rejecting Test Cases. TTLs should always open each Test Case submission and check for all requested materials before choosing to Approve or Reject. Please consider the following points in triaging test cases:

    • a. Did the tester include all requested documents or information with the Pass/Fail verdict?
      • i. If attachments were requested, confirm that there actually is an attachment. Then be sure to open them and verify they are in the correct format and contain the necessary information. If multiple submissions were made by a single tester, compare these documents across the various completed test cases to verify they are original and not simply duplicates with different titles and dates.
      • ii. If the tester was asked to do exploratory testing, make sure the exploratory notes are filled out completely and cover a sufficient amount of testing time dependent on the expectation set in the overview or based on the payout (i.e. a $10 test case will probably have a lower expectation of work than a $30 test case).
    • b. Were the testers required to sign up (usually via an external document) for a limited number of test cases? If so, check to be sure the tester’s submission you are approving is slotted for a test case and that the device/browser/etc. details match.
      • i. If the testers were required to sign up, were they instructed to only fill a specific number of slots? Keep an eye on testers who try to reserve more than their share of test case slots.
  4. Bug Fix Verification
    If the TTL is working with a customer who either has bug tracking integration with our platform or for whom you are manually transferring our defects, you may also be responsible for identifying the need for (and possibly kicking off) Bug Fix Verification (BFV) cycles. Clarify with the PM who will be responsible for BFV management before the test cycle starts. Keep an eye on the fix statuses of defects that have been moved from the uTest platform to the customer’s bug tracking system and either manage the BFV cycles in a timely fashion or notify the PM which issues need attention. If a bug is verified fixed by the testers via BFV, be sure to update the status to the corresponding bug in the customer’s tracking system. If the testers find that that the issue is still recurring, report this status in the other system and continue to monitor it there, kicking off another round of verification testing when warranted. Again, be sure to clarify with the PM who will be responsible for BFV management before the test cycle starts.
  5. Disputed Bugs
    If a bug is disputed by the tester, the TTL should review the comments and then speak to tester on Tester Messenger to notify the tester that their bug is being reviewed. If the TTL has control of rejections and approvals, they can determine a final status. If rejected, the bug changes to Under Review for the PM to make a final determination. If the TTL made a mistake in the initial triage of the bug, they can recommend to the PM that the dispute be resolved and the bug approved.
  6. TTL Testing on Test Cycles
    TTLs should discuss whether they can test on the same test cycle that they are TTLing with the PM for that test cycle before launch. If the PM agrees that the TTL can test on the test cycle, then the TTL should notify the testers through the welcome message in chat to reduce confusion for the testers.

Expectations and Escalations

The following list contains the expectations and escalation paths specific to all TTLs. The tester Code of Conduct applies in all cases for all testers and TTLs.

  1. Expectations: TTLs are expected to…
    • Respond as quickly as possible to questions in chat for the test cycle. The availability schedule for the TTL and the PM should be listed in the Scope and Instructions. While testers typically expect chat responses within an hour of submission, TTLs should attempt to respond with 2-4 hours during normal business hours and within 12 hours during the nights and weekends.
    • Closely monitor test cycles directly after it launches for questions in chat and issues with bug reports.
    • Monitor chat for appropriate tester behavior and escalate any issues that are observed.
    • Help reduce the noise created by testers and reduce the number of information requests that PM’s and TM’s must address by gaining a strong understanding of the application and the scope of the test cycle and taking the lead role in answering questions in chat or in the defects.
    • Triage defects and test cases during the test cycle. Feedback (including approvals, rejections, or requests for more information) to testers during the test cycle can result in more detailed information in the bug reports and higher quality bug reports and test cases available to the customer. This increase in response to testers can assist in improving the overall quality for subsequent bugs in the current test cycles and future test cycles.
    • Provide feedback to testers based on their knowledge of the test cycle, the application under test, the customer, the devices under test, and the expectations listed by the PM prior to test cycle launch. It is understood that this information may change test cycle to test cycle or even within a single test cycle.
    • TTLs should provide feedback to testers when expectations are not met in defects, test cases, and reviews. They may also provide feedback for exceptional work.
    • Be flexible, very flexible.
      NOTE: If a TTL is unable to commit to the expectations of a particular test cycle they must give the Project Manager sufficient notice so that the role can be successfully backfilled. Sufficient notice will vary by test cycle, at the very least a TTL must give 48 hour notice that they will be unable to perform the work required. However, if there is a longer on-ramping time associated with a cycle, then the TTL should inform the PM two weeks prior to exiting the role of TTL.
  2. Escalations:
    1. For any issues dealing with the test cycle, the TTL escalates to the primary Project Manager first. If there’s not a response within 24 hours for normal issues or 12 hours for urgent issues, then the TTL should send an email to PM.Escalate@utest.com with a subject line in the following format – [TTL Escalation for Customer Name and Test Cycle ID#] and details on the issue in the body of the message. A TTL should only directly contact the customer if permission has been granted by the Project Manager, either at the project or test cycle level.
      Example 1: Testers record questions in chat about the fact that there is no review tab active for the test cycle, though the scope refers to customer reviews when testing is completed.
      Example 2: For an e-commerce site, testers are asking whether the entire shopping cart function is out of scope or only the final purchase step.
    2. Any issues with testers behavior, Code of Conduct infractions, or otherwise should be escalated to CM with notification to the PM.
      Example 1: Tester discusses test case pay out rates in chat.
      Example 2: Tester disputes rejected bugs multiple times.
      Example 3: Tester misuses slotting spreadsheets.
    3. As mentioned above, if a TTL is unable to commit to the expectations of a particular test cycle they must give the Project Manager sufficient notice so that the role can be successfully backfilled. Sufficient notice will vary by test cycle, at the very least a TTL must give 48 hour notice that they will be unable to perform the work required. However, if there is a longer on- ramping time associated with a cycle, then the TTL should inform the PM two weeks prior to exiting the role of TTL.

TTL Bill of Rights

  • Receive notification of test cycle assignments prior to launch of the test cycle
  • Receive anticipated scope of work prior to TTL accepting each TTL assignment
  • Receive information about anticipated payout prior to TTL accepting each TTL assignment
  • Receive notification from the PM prior to the launch of the next test cycle if PM decides to swap TTLs for any reason
  • Receive a response from the PM within 24 hours for normal issues or 12 hours for urgent issues during the following timeframe: 8am – 8pm US Eastern time. If you don’t hear back, you may email to PM.Escalate@utest.com with the customer name and test cycle ID in the subject line
  • Expectation that testers will contact TTLs through the platform only, not through any other method like Skype, IM, Google+, etc.

uTest Business Model

For TTLs who have not experienced uTest for a long time, here is a brief outline of uTest’s business model. This knowledge should put things into perspective when you have outstanding questions. However, the Community Management Team is always available to address any open questions about the uTest business model and how it pertains to any TTL assignments – past, present, or future.

The uTest business model is built on these Customer Expectations – that uTest:

  • Provides consultation for test strategy, test cycle configuration, and test cycle optimization to achieve high-quality results
  • Provides various encapsulated industry and domain-level knowledge and experience
  • Provides testing expertise and diverse testing coverage
  • Provides high value defect reports
  • Responds quickly to customer and testers alike
  • Assures a low level of noise during each test cycle

Basic Business Concept:

  • Larger customers generally engage uTest for longer timeframes to ensure continuity of testing coverage and expertise. In general, Project Managers that cover larger customers have fewer total customers, as this allows for deeper focus on customers who generally have greater needs.
  • Smaller customers may engage uTest for a variety of time periods, whether it’s several test cycles or annual contracts. In general, Project Managers that cover smaller customers will have more customers, which generally results in slower response times.

Key Message for TTLs: No matter the size of the customer or the size of the test cycle, every customer should be delighted with the test cycle experience from activation to close.

TTLs and SDLC

A basic understanding of the Software Development Lifecycle (SDLC) is very useful for every TTL. The more confidently that a TTL understands the different stages of the SDLC the better that the TTL can prepare and set appropriate expectations for the testers on the test cycle. In general, applications that are tested earlier in the development cycle (like an alpha release) have code that is relatively untested, and testers can expect to find more bugs. On the other hand, applications that are being tested later in the development cycle will have more stable code with fewer bugs that are more difficult to determine and more critical to the live release of the software. Please refer to the Wikipedia definition of the SDLC to begin your research on this topic.

Conclusion

The term TTL is a uTest-derived acronym that stands for ‘Test Team Lead.’ In a Test Cycle, a TTL is the bridge that connects Project Management, Testers, Customers, and Community Management and assists in increasing the uTest value to the customer and the testers.

TTLs have 3 primary tasks in a test cycle:

  • Platform communications
  • Triaging Bugs
  • Triaging Test Cases

Be sure to confidently understand the standards and expectations listed in this handbook before engaging in any test team leadership activities. Because it cannot be repeated enough, the goal for the TTL is to increase the value that uTest brings to the customer by maximizing the output and minimizing the noise level. As a TTL, we are placing a great deal of trust in your technical and leadership abilities to help manage each test cycle. Therefore, please do not hesitate to reach out to the CM Team if you do not believe we are providing appropriate training or expectations for all parties involved in order for you to become successful and for our customers to extract the highest value from uTest. Lastly, feel free to reach out to the uTest Community Management Team if you have any questions or concerns with current or recent TTL assignments.

Recommended Next Steps

Once you feel comfortable with the information contained within the TTL Handbook you may request access to the TTL Entry Level Exam by emailing TTL@utest.com.

  • Participate in on-line training sessions as they are available.
  • Further training may be required:
    • Rules about exporting bugs:
    • CSV
    • Bug tracking systems
  • Communications with PMs
  • Communications with CM

Addendum

Chat Examples:

Example: — New Chat Topic – Welcome message:

When the test cycle launches, the TTL should introduce themselves as the Test Team Lead for this test cycle in the Chat Topics. This notification establishes a point of contact testers can reach out to with their questions and observations. A generic welcome message is provided below. Amend as appropriate:

Welcome to this<Project name> cycle. I will be your Test Team Lead on this project. At any time, please let me know if you have any questions or concerns. I will try and find an answer for you as soon as possible.

Please ensure that you read the scope and instructions carefully. Check the Out of Scope section and known issues list before starting your testing effort.

In addition, make note of other bugs raised during the cycle to assure that you do not raise duplicate bugs.

When raising issues, please ensure that the bug titles and descriptions are of the highest quality. They should be descriptive and clear. The bug title in particular should be descriptive enough to explain the area and issue so it is easy to spot when going through the bug list.

Please be sure to include all mandatory information for the bug as required.

Thanks a lot and happy testing

<TTL’s Name>

Goodbye Ryan

Back in June 2015, my good buddy an uTest Community Manager extraordinaire Ryan Arsenault decided to leave Applause in pursuit of other opportunities.

Ryan wrote a farewell post to the uTest community that showed what a classy guy he is; always willing to help or just chat with testers, even late in the evening and always working to keep our testers engaged and improving.

Thanks Ryan, I’ll miss you!

 

2014 Year in Review

I’d love to write a witty post recapping the last year, but right now I just don’t have the time. So a brief summary will have to do for now.

Job Change

In February, I decided to leave Brock Solutions and work for uTest as a full-time consultant. I spent the next 8 months working for SalonCentric (a L’Oréal devision) as a business/test manager as they built and launched their new eCommerce website. That was wonderful time. I make some fantastic friends and learned a lot about testing eCommerce, particularly on the Demandware platform.

I was promoted to an Enterprise Test Team Lead which is essentially a contracted Test Project Manager. I was assigned several customers and project all of which were delivered successfully and received praise and accolades.

uTest

University – I wrote or co-wrote several courses for the uTest University

Forums – I stayed quite active on the forums as usual. I’m almost at 2,000 posts.

  • I started a really fun series titled Analyze This where I posted a testing “truth” and we debated it. Although technically I started this in 2013, the majority of the posts happened in 2014. So far I’ve started 13 topics, all of which earned at least 2,000 views and several are approaching 10,000!
  • I also made a very controversial proposal to change the uTest rating system to eliminate rating hits. There’s a lot at stake so I understand uTest’s hesitance to move on it, but I still think this is the right thing to do.

Blog – I wrote a few posts for the uTest blog, two of which were selected for The Best of 2014: Top Posts from the uTest Blog. “Certified and Proud?” was one of uTest’s most popular posts of the year, earning the largest number of comments and social shares. Even a few A-list testers read and talked about it!

A.C.E – uTest started a new initiative to provide testers with organized, mentor-based training. I helped put together the pilot program and was one of the mentors for it. A large portion of the materials we used were things that I’ve written for uTest over the years. I’m now working on converting all of my previous articles and papers into more structured courses as the A.C.E. program continues to evolve and improve.

Awards – uTest created a Hall of Fame website and introduced quarterly awards based on votes from the community. In the two quarters that this as been going on, I’ve won 3 awards

  • Outstanding Forums Contributors
  • Outstanding Blogger
  • Outstanding uTest University Instructor

AST-BBST

In September, I successfully completed the Foundations Course. That was quite an experience. I hope to find some time to write about it one day.

In November, I passed the Instructors course and will be an assistant instructor for the first Foundations class of 2015.

uTest University Webinar – Live Exploratory Testing

To go along with my course on Exploratory Testing, I had the opportunity to do a live webinar demonstrating how I do exploratory testing. The community management team did some great marketing and over 300 people signed up! Unfortunately, we found out at the start of the webinar that the uTest GoToMeeting license only supported 100 participants so a lot of folks were not able to attend. It was quite a humbling experience to have so many people interested in watching how I test.

As you can see from the video, the app I chose was quite buggy. That coupled with my poor internet connection presented some interesting challenges. It was difficult to maintain my train of thought and to investigate a single issue when other issues kept popping up, but I think it gave a good representation of what exploratory testing can be like sometimes.

Looking back at the video I need to improve in the following areas:

  • Speak slower and clearer – I was trying to articulate my every thought and this lead me to speak in quick, incomplete sentences. It might have been beneficial in showing how choppy my thought process can be, but I think it might also made it difficult to follow what I was thinking.
  • Repeat the question – During the Q&A portion, I didn’t do a good job of stating the question I was answering. I read the question and kinda mumbled it before jumping into the answer. I did OK on some questions, but I need to remember to do a better job of this next time.

http://university.utest.com/recorded-webinar-exploratory-testing-lucas-dargis/

My Testing bucket list

I realize this might appear to be a vain post, but the intent is to set some goals for myself and make them public in order to help me be more motivated to achieve them. I’ll probably be adding/removing items as my career progresses and my goals change.

  • Win a uTest tester of the year award
  • Mentor a new uTest TTL until they become an ETTL – Prashanti
  • Present at a testing conference
  • Have an A-list tester know who I am
  • Complete all the AST-BBST courses
    • Foundations course
    • Bug Advocacy
    • Test Design
    • Instructors
  • Instruct a AST-BBST course
    • Assistant instructor – Foundations 101-AC (January 2015)
    • Lead instructor
  • Publish an article in various testing publications/blogs
    • http://blog.utest.com/
    • http://www.testingcircus.com/
    • http://www.softwaretestpro.com/

 

 

uTest blog post – Three Things Testers Can Learn From Wine Experts

Originally posted on the uTest blog: http://blog.utest.com/2014/08/22/three-things-testers-can-learn-from-wine-experts/

Also posted here: http://blogs.yourzephyr.com/?p=3987

When I’m not testing, one of my favorite hobbies is alcohol. Wait…that didn’t come out right. What I meant was my hobby is learning about winSommelier_e_Tastevine, beer and sprits. Yeah, that sounds better.

While I do love a cold beer in the summer, a single-malt scotch when I’m feeling sophisticated, or an 1855 classified Bordeaux on special occasions, I think I spend more time studying booze than I do drinking it. I really enjoy learning about the various Appellation d’origine contrôlée (AOCs) in France, and the differences between a Pinot Noir from California and one from Burgundy. I sound pretty smart, huh?

As any cultured, refined wine connoisseur such as myself knows, the true masters of the bottle are called sommeliers. These fine folks are highly trained adult beverage experts who often work in fancy, fine-dining restaurants, setting the wine list, caring for the cellar and working with customers to help them select the perfect wine.

So what could a tester possibly learn from a someone obsessed with booze? Good question! I have three answers.

Be passionate

I have yet to find people who are more passionate about what they do than Master Sommeliers. Need proof? Watch the movie Somm (available on Netflix). The tremendous amount of dedication and effort these people pour (wink, wink) into their work is simply astounding.

A sommelier must be constantly learning and exploring. Each year, a new vintage of every wine is created. That means thousands of new wines are added to the multitude that already exist…and a sommelier is expected to be familiar with with all of them. And you thought the IT world was constantly changing!

There will always be a new product to test, a new approach to learn, a new idea to debate. Testers who are passionate about testing are excited about these new developments as they are opportunities to grow and improve.

Be a servant

From the Demeanor of the Professional Sommelier:

It is important for sommeliers to put themselves in the role of a server; no job or task on the floor is beneath the role of a sommelier; he or she does whatever needs to be done in the moment to take care of the guest.

Sommeliers are at the top of the service food chain. They are highly trained, knowledgeable, and professional people, yet they are also the most humble servants. They realize that their qualities alone are worthless. They must be used in the service of a customer for their value to be realized.

Testers too need to remember that we don’t get paid because we think we’re great testers. We get paid because the person who is paying us values our work. Put your client’s perception of value and quality ahead of your own.

Be an expert

A sommelier would never walk up to your table and say, “I recommend this bottle I found in the back.” They are the ultimate experts in wine selection and food pairing. He or she asks questions: What do you like? What have you had before? What is your budget? They learn about the customer and use that information to help them find exactly the right bottle.

Likewise, a tester should be knowledgeable in the testing field. A good tester doesn’t just randomly go banging on things — they too take a more thoughtful approach. What areas are the most error prone? What parts of the product are the most important? What information would be most useful at this stage in the product’s life cycle? They learn about the product and the circumstances under which they are testing to ensure they provide the most value possible.

Take pride in your work. Understand that testing is not a low-skilled job; it is a highly cognitive profession with demands on your professionalism, communication skills, and attention to detail. It takes a lot of effort, study and experience to become an expert (or so I’m told), but that should be the goal of every tester.

uTest blog post – Certified and Proud?

I recently wrote a post for the uTest blog about my certification story. It was well received earning a good number of comments and social shares.

http://blog.utest.com/2014/08/04/certified-and-proud-a-testers-journey-part-i/
http://blog.utest.com/2014/08/05/certified-and-proud-a-testers-journey-part-ii/

 

A Gold-rated tester and Enterprise Test Team Lead (TTL) at uTest, Lucas Dargis has been an invaluable fixture in the uTest Community for 2 1/2 years, mentoring hundreds of testers and championing them to become better testers. As a software consultant, Lucas has also led the testing efforts of mission-critical and flagship projects for several global companies.

Here, 2013 uTester of the Year Lucas Dargis here shares his journey on becoming ISTQB-certified, and also tackles some of the controversy surrounding certifications.

In case you missed it, testing certification is somewhat of a polarizing topic. Sorry for stating the obvious, but I needed a good hook and that’s the best I could come up with. What follows is the story of my journey to ISTQB certification, and how and why I pursued it in the first place. My reasons and what I learned might surprise you, so read on and be amazed!

Certifications are evil

Early in my testing career, I was a sponge for information. I indiscriminately absorbed every piece of testing knowledge I could get my hands on. I guess that makes sense for a new tester — I didn’t know much, so I didn’t know what to believe and what to be suspicious of. I also didn’t have much foundational knowledge with which to form my own opinions.

As you might expect, one of the first things I did was look into training and certifications. I quickly found that the pervasive opinion towards certifications (at least the opinion of thought leaders I was learning from) was that they were at best a waste of time, and at worst, a dangerous detriment to the testing industry.

In typical ignoramus (It’s a word, I looked it up) fashion, I embraced the views of my industry leaders as my own, even though I didn’t really understand them. Anytime someone would have something positive to say about certification, I’d recite all the anti-certification talking points I’d learned as if I was an expert on the topic. “You’re an idiot” and “I’d never hire a certified tester” were phrases I uttered more than once.

A moment of clarity

Then one fine day, I was having a heated political debate with one of my friends (I should clarify…ex-friend). We had conflicting views on the topic of hula hoop subsidies. He could repeat the points the talking heads on TV made, but when I challenged him, asking prodding questions trying to get him to express his own unique ideas, he just went around in circles (see what I did there?).

Like so many other seemingly politically savvy people, his views and opinions were formed for him by his party leaders. He had no experience or expertise in the area we were debating, but he sure acted like the ultimate authority. Suddenly, it dawned on me that despite my obviously superior hip-swiveling knowledge, I wasn’t that much different from him. My views on certifications and the reasons behind those views came from someone else.

As a tester, I pride myself on my ability to question everything, and to look at situations objectively in order to come to a useful and informative conclusion. But when it came to certifications, I was adopting the opinions of others and acting like I was an expert in an area I knew nothing about. After that brief moment of clarity, I realized how foolish I’d been, and felt quite embarrassed. I needed to find the truth about certifications myself.

Now let me pause here for a second to clarify that I’m not saying everyone should go take a certification test so they can have first-hand experience. I think we all agree that it’s prudent to learn from the experience of others. For example, not all of us have put our hand in a fire, but we all know that if you do, it will get burned.

It’s perfectly fine to let other people influence your opinions as long as you are honest about the source of those opinions. For example, say:

Testers I respect have said that testing certificates are potentially dangerous and a waste of time. Their views make sense to me, so it’s my opinion that testers should look for other ways to improve and demonstrate their testing abilities.

However, if you’re going to act like an expert who has all the answers, you should probably be an expert who has all the answers.

Studying for the exam

I took my preparation pretty seriously. Most of my studying material came from this book which I read several times. I spent a lot of time reading the syllabus, taking practice exams and downloading a few study apps on my phone. I also looked through all the information on the ISTQB site, learning all I could about the organization and the exam.

Since I knew all the anti-certification rhetoric about how this test simply measures your ability to memorize, I memorized all the key points from the syllabus (such as the 7 testing principles). But I also took my studying further, making sure I understood the concepts so I could answer the K3 and K4 (apply and analyze) questions.

Headed into the test, I felt quite confident that I was going to ace it.

Be sure check out the second part of Lucas’ story at the uTest Blog.

————————————————————————————————-

his is the second part of tester and uTest Enterprise Test Team Lead Lucas Dargis’ journey on becoming ISTQB-certified. Be sure to check out Part One from
yesterday.

The test

After about 3 minutes, I realized just how ridiculous the test was. Some of the questions were so obvious it was insulting, some were so irrelevant they were infuriating, and others were so ambiguous all you could do was guess.

Interestingly, testers with experience in context-driven testing will actually be at a disadvantage on this test. When you understand that the context of a question influences the answer, you realize that many of the questions couldn’t possibly have only one correct answer, because no context was specified.

You are allotted 60 minutes to complete the test, but I was done and out of the building in 27 minutes. That I finished quickly wasn’t because I knew all the answers — it was rather the exact opposite. Most of the questions were so silly, that all I could do was select answers randomly. Here are two examples:

“Who should lead a walkthrough review?” – Really? I was expected to memorize all the participants of all the different types of meetings, most of which I’ve never seen any team actually utilize?

“Test cases are designed during which testing phase?” – Umm…new tests and test cases should be identified and designed at all phases of the project as things change and your understanding develops.

According to the syllabus, there are “right” answers to all the questions, but most thinking testers, those not bound by the rigidness of “best practices,” will struggle because you know there is no right answer.

Despite guessing on many questions, I ended up passing the exam, but that really wasn’t a surprise. The test only requires a 65% to pass, so  a person could probably pass with minimal preparation, simply making educated guesses. I left the test in a pretty grumpy mood.

 

The aftermath

For the next few days, I was annoyed. I felt like I had completely wasted my time. But then I started thinking about why I took the test in the first place, and what that certification really meant. As a tester striving to become an expert, I wanted to know for myself what certifications were about. Well — now I have first-hand experience with the process. I’m able to talk about certifications more intelligently because my opinions and views about them are my own, not borrowed from others.

Former tennis great Arthur Ashe once said, “Success is a journey, not a destination. The doing is often more important than the outcome.” I think this sums up certifications well. For me, there was no value in the destination (passing the exam). I’m not proud of it, and I don’t think it adds to or detracts from my value as a tester. However, I felt there was value in the journey.

Through my studying, I actually did learn a thing our two about testing. I was able to build some structure around the basic testing concepts. I learned some new testing terms that I have used to help explain concepts such as tester independence. Studying gave me practice analyzing and questioning the “instructional” writing of others (some of which I found inaccurate, misleading or simply worthless). The whole process gave me insight into what is being taught, which helps me better understand why some testers and test managers believe and behave the way they do.

But more importantly, I became aware of the way I formulate opinions and of how susceptible I am to the teachings of confident and powerful people.

The lessons I learned from testing leaders early in my career freed me from the constraints of the traditional ways of thinking about testing, but in return, I took on the binds associated with more modern testing thinking. Ultimately, I came to a conclusion similar to that of many wise testers before me, but I did so on my terms, in a way that I’m satisfied with.

One day, I hope to have a conversation with some of the more boisterous certification opponents and say, “I decided to get certified, in part, because you were so adamantly against it. I don’t do it out of defiance, but rather out of a quest for deeper understanding.”

If you made it this far, I thank you for sharing this journey with me. Make a mention that you finished the story in the Comments below, and I’ll send you a pony.

How to Write a Quality Bug Report

I worked with Aaron Weintrob to put together a uTu course on how to write quality bug reports. The intended audience of this course is new uTesters with little or no testing experience. We kept it relatively short and simply highlighted the key areas. We may create additional courses for various sections where we can dig in a little deeper. Here is a link to the course. Below is the course’s content:

INTRODUCTION

Writing a good bug report is one of the most talked-about topics in the testing world. The art of creating a well-written bug report requires a balanced combination of testing and communication skills. This course provides advice and tips geared towards helping you create bug reports that are informative and actionable, thus improving their value to the customer.

WHAT MAKES A GOOD BUG REPORT?

Most testers understand the role of a bug report is to provide information, however a “good” or valuable bug report takes that a step further and provides useful information in an efficient way.

To help us get started writing valuable bug reports, we are going to focus on a few key areas:

  • The Title
  • Actions Performed (Steps)
  • Expected and Actual Results
  • Attachments

THE TITLE – THE GOOD, THE BAD, THE UGLY!

The title is the face of your report. It’s the first thing anyone sees and it’s importance cannot be overstated. A good title helps reduce duplicate issues and can quickly convey a summary of the bug.

It’s best to avoid generic problems in the title. For example, these should never be used:

  • XYX is not working properly
  • Issue with XYZ
  • XYZ is corrupted/does not look right

The above example titles add little value in describing the problem. By nature, every report is describing something that is not working as it should. Be specific about what makes it “not working.”

Instead of: Sorting is not working properly.
Try: Sorting is happening in reverse order.

Instead of: Issues with GUI on navigation bar.
Try: Navigation bar is wrapping to a second line.

Often times, bugs are migrated into the developer’s database that may contain hundreds, if not thousands, of other issues. Imagine trying to search this database for “navigation bar”. That search will return every issue related to the navigation bar. Searching for “wrapping to second line” is much more specific making it easier to find the bug. Your bug report needs to survive (and be useful) beyond the current test cycle; a strong title will help it through it’s journey.

ACTIONS PERFORMED – ADVICE FOR EXPLAINING YOUR STEPS

This is the body of your report. The goal of this section is to show the reader how to reproduce the bug. Since this area usually contains the most information, it’s important to keep it concise and easy to read. Always number your steps and kept them short and to the point.

Tip: Using a prerequisite can reduce the number of steps.
Instead of listing out every step to login in, start your steps with: “Prerequisite: User is logged in”

Tip: Find the direct path to the bug
Often times, testers will stop at the point where they found a bug and log their last few actions. However, the most helpful bug reports are those that distill the report down to the core reproduction steps.

It’s a good exercise to reproduce the bug by following the steps you’ve just outlined. This will help ensure you’ve included everything the customer will need to reproduce it as well.

Sometimes digging a little deeper below the surface of the bug can add additional value. Here are some examples of how adding a bit more effort or thought will produce a higher quality report.

Example 1: Provide additional useful information
Scenario: You find that a video does not play.
Good: Mention if it happened on all videos and not just the one mentioned in report.
Better: Specify if the issue is reproducible on more than one browser or device.
Best: Upload a speed test showing that bandwidth was adequate when testing was happening.
Lesson: Try to identify and answer follow-up questions before the customer asks them.

Example 2: Report the bug, not a symptom of the bug
Scenario: We are testing an Address input field. We find that the Address field allows “1234567890″ and it also allows “!@#$%^&*()_+”
Lesson: These are two different symptoms of the same bug. Closer inspection would reveal that the real issue is the Address field isn’t being validated at all. The problem may be more serious than the first symptom you find.

EXPECTED AND ACTUAL RESULTS – WOULDA, COULDA, SHOULDA

Now that you have described how to reproduce the bug, you need to explain the problem and the desired behavior.

Tip: When describing expected results, explain what should happen, not what shouldn’t happen.
Instead of: The app shouldn’t crash.
Try: The user is taken to XYZ screen.

Tip: When describing actual results, describe what did happen, not what didn’t happen.
Instead of: The user wasn’t taken to the XYZ screen.
Try: The user remained on the ABC screen.

ATTACHMENTS – WHAT TO DO AND WHAT NOT TO DO

Attachments add to the bug’s value by offering proof of the bug’s existence, enabling the customer to reproduce it or helping the developer fix it. Each attachment should add to the value of the bug in at least one of these three ways.

The following are some tips and guidelines to keep in mind when adding attachments:

IMAGES

  • Adding images is a quick way to add context to your bug. Consider adding an image even if you also have a video.
  • Highlight the area(s) of interest in your image.
  • Attach the image files directly to the report. Don’t put images in a Word document or a zip file.
  • Use images to illustrate static issues.

VIDEOS

  • Video confirms your steps were accurate at the time the issue was created. For example, a screen grab of an error message isn’t as useful as seeing what went into creating that error message.
  • Actions in the video should match the steps listed in the bug report.
  • Videos should be trimmed to only show the bug.
  • Provide video if the steps are complex.
  • External/live videos can be more impactful than mirrored videos because you can see hand gestures or you touching a button on the screen.

LOG FILES AND OTHER TIPS

Avoid proprietary file types (like .docx). Use .txt instead.
Avoid compressed (.zip) files unless specifically asked for or approved by the TTL, PM, or customer.

UTEST ETIQUETTE – GENERAL OVERVIEW OF PROPER BEHAVIOR

It is important to remember that you are representing the TTL, the test team, and all of uTest when you work on the test cycle. Your fellow testers rely on you to write a good title for your bug report so they won’t file a duplicate bug. TTLs depend on clean, good reports to ensure the customer receives value from the cycle. uTest needs quality work from everyone so we can continue to work in the field we all love.

ADDITIONAL READING

Here are some valuable discussions about bug reports from the uTest Forums:

My First Testing Experience – Part 2

Continued from Part 1

I can vividly remember lying on my couch in my living room staring at the ceiling, my stomach in knots. I felt this huge weight on top of me making it hard to breath. I was pretty close to panicking. “How can I possibly do this? I have no idea what I’m doing. Why did I agree to this project? Do they have any idea what they’re asking? This is so unfair. This is CRAZY!” These were the thoughts racing through my mind. Before I could think one through, another would jump on the pile. That feeling of hopelessness, of being completely overwhelmed was my first and worst moment as a software tester.

I’d like to think the PM understood the magnitude of the project, that somehow she saw this amazing potential inside me just waiting for an opportunity to prove itself. But I knew she didn’t really understand what she was asking. In her mind this was a small task that any mid-level developer should be able to do. After all, it’s just testing, how hard can that be? That just added to the pressure. I was dealing with unrealistic expectations about an undefined project that required an under-appreciated amount of skill and effort.

After breathing into a paper bag for 10 minutes or so, I calmed down. I reminded myself the best way to climb a mountain is one step at a time. I started by making a spreadsheet documenting all the different migration tools we had built. I identified the base data type, the destination data type and a description of the transformation needed. One by one I went down the list, talking to the developers, searching for any documentation, looking at the old system, learning about the new one, basically doing the work a BA would have done.

Once I had a solid understanding of the specifications I was working with, I started testing. I started with the simplest cases and the ones that I was most familiar with. You are probably thinking I should have done the opposite and started with the hardest, most complicated and risky transformations. If I was doing it over today, that is the approach I would take, but considering I was a complete rookie tester, I needed to start with a small, achievable task to build some confidence and establish my testing strategy.

My strategy was pretty simple. I gathered sample source data, ran it though the migration and verified the results on the other side. Then I started adjusting the source data, trying every scenario I could think of looking for ways to make the tools fail. As I worked through my list, reporting bug after bug, it became clearer to everyone just how large this task was. Expectations slowly became more aligned with reality and the measurement for success became more reasonable.

After a few months of work, we were ready to start migrating content. It wasn’t perfect and several bugs made it through, but they were minor and due to junk source data that we didn’t care about. The migration ran for months as different sites came online. The migration tools did their job and by all accounts, the testing aspect of the project was successful.

This experience gave me an unexpected appreciation for what testing is and how valuable it can be. It gave me confidence in myself that I could accomplish a task that I really had no business even attempting. It’s an experience that I can now look back on and be proud of.