Tag: Education

uTest TTL Handbook

The Fundamentals for All Test Team Leads

This handbook explains the basic concepts surrounding the uTest Test Team Lead role.

Developed by uTest Community Management  – 2/11/2014

Introduction

The term TTL is a uTest-derived acronym that stands for ‘Test Team Lead.’ TTLs are hand-selected members of the uTest community who work side-by-side with uTest Project Managers (PMs) to facilitate the flow and outcome of test cycles. A few traits and required skills of good TTLs include:

  • Hard skills: high-quality testing output with proven track record, attention to details, expertise managing test cycles based on customer requirements across multiple technologies, Experience with uTest and an understanding of test cycle operations, and device/industry specific expertise
  • Soft skills: excellent communication skills, professionalism, culturally sensitive, fair and helpful
  • Sense of urgency: attentive to test cycle details and activity, focuses on responsiveness, escalates issues accordingly to the PM or CM, constructive not confrontational
  • Business acumen: understands what’s important to the customer with the ability to analyze and communicate the information across the testing community when necessary

Although the tasks required per test cycle may differ, the goal is the same: to increase the value that uTest brings to the customer by maximizing the output and minimizing the noise level.

Distribution of TTL Work

A TTL is vetted and trained before being placed onto paid projects. Paid projects are assigned by a PM. TTL payouts are always defined upfront, before any work is performed, and are kept between the TTL and the PM. Each assignment is different and depends on many factors, such as the TTL’s ability to triage bug reports or their ability to create test cases, to name two. Other criteria may include the TTL’s experience in a particular testing domain, industry, or mobile platform and access to those platforms to reproduce bugs. TTLs should utilize the TTL Availability Google Doc to enter your weekly availability and project preferences. Request access through the link provided.

Details of TTL Assignments

For many test cycles, the TTL’s main responsibilities are to monitor tester questions via test cycle chat and triage reports as they come in. TTLs should be in regular contact with the PM to ensure that the test cycle is headed in the right direction and that the customer is delighted with the results. Below are details of the responsibilities TTLs may be asked to perform:

  1. Platform Communications with Testers and Customers
    No matter the type of communication, keep in mind that testers must follow the uTester Code of Conduct at all times. TTLs should monitor the chat as closely as possible in order to address incoming questions and deliver outgoing announcements, especially in the first few hours that the test cycle launches.

    • Chat: Test cycles typically have real-time chat enabled, which allows TTLs to communicate with testers, customers, and project managers in real time. There are three components of chat: Questions about Scope, Announcements and Threaded/Topic Conversations.
      • Questions on Scope – for general questions on the cycle. This is usually related to scope questions, but can contain other general questions about the cycle such as questions on difficulty installing the build or claiming a test case.
      • Announcements – where TTLs should place their welcome message to the testers (see example welcome message in the addendum). Also, this is where the TTL would make any other announcements such as: announcing a new build, announcing a new out of scope section, making a correction to the information in the overview, reminding testers about the cycle close time, encouraging testers to get test cases submitted, refocusing the team on the focus areas.
      • Topic Conversations – discussions isolated to a specific topic. If there is a major development that the TTL thinks should have more attention than just an announcement, consider starting a new thread. These Topics can garner more attention because the TTL can edit the title. Threaded conversations are also good for topics like test case claims. Using these topics can keep the Questions about Scope section a bit cleaner.
        NOTE: All chat activities can result in an email sent to the testers and the customer depending on their settings, so please be mindful of the noise that chat can create.
    • Tester Messenger: Notes that TTLs add to Tester Messenger will result in an email directly to the tester who reported the issue and is added to the defect report for easy reference. This message is visible to the customer, PM, and any other testers on the test cycle. TTLs should use the Tester Messenger when:
      • There is a question about the defect that requires an answer from the reporting tester.
      • The testers will sometimes leave a message here for the TTL or the customer when the TTL, PM or customer leaves them a message in the tester messenger. Be aware that testers cannot initiate that exchange. In addition, the TTL receives an email any time a message was added to a bug that they have commented on. When testers leave a message for the customer, the customer should have visibility and may respond, but if historically they are not that responsive, the TTL can email the Project Manager, relaying the question, then respond to the tester in the Tester Messenger.
        NOTE: Be careful about exposing the TM’s contact information to the tester if responding to the tester via email (say, forwarding the response). The TTL is never to give out the TM information unless expressly asked to do so by the PM. If so, please send the information to the tester privately through email or chat to keep that information out of the tester messenger where it could be seen by other testers. It is preferred for uTest to be the go-between rather than to start opening doors between the testers and TMs.
      • If the question and answer are determined to be valuable to other testers by the TTL, share this information on the Chat/Scope & Instructions with all the testers.
    • Request More Info: If there is missing information or more clarification is required from a tester, the TTL can push a bug report to the “Request More Info” status, which shows that the bug report was reviewed but requires a response from the tester. Be aware that CM recommends using this status only for small portions of missing information. For a bug that has more significant issues, for example it does not follow the customer’s bug template, is clearly a placeholder, or is just poorly written, the bug should be rejected as “Did Not Follow Instructions.” TTLs are encouraged to assist the overall community in elevating the expectations of testers and eliminating consistently poor bug documentation performance by rejecting bugs that meet those criteria.
      NOTE: Sending reports into the “Request More Info” status automatically adds a note to the Tester Messenger and prevents testers from discarding these bugs. However, it is possible to visually note when a tester has responded through the Platform UI and through email notification as the status of the bug returns to “New”. In addition, this status assists the testers when they see their bug change status. Testers should be reminded to press “Send Requested Info” button. Please note that if the tester does not respond before the cycle closes, then their bug will be automatically rejected.
    • Customer Notes: Notes that you add to Customer Notes sends an email to the customer’s Test Manager and makes a copy of the message in the defect report for easy reference. This message is visible to the customer, PM, and any TTLs on the test cycle who have TTL permissions (but not to testers). TTLs should use the Customer Messenger when:
      • The defect needs to be exported to the customer’s tracking system
      • The defect has been exported to the customer’s tracking system and the TTL needs to leave a note to identify the corresponding defect ID# in the other system.
        NOTE: The key to keeping track of what needs to be exported versus what has already been exported is to be consistent. For example, the TTL leaves a Customer Note only when the defect has been exported to the customer’s tracking system. Then in that note, simply enter the corresponding defect ID# from the other system. If all of the defects will not be exported to the customer’s tracking system, or if the defect is being approved but is not being exported to the customer’s tracking system, the TTL can insert a message to denote that.

        • The TTL needs to communicate directly with the customer about this issue, such as to provide clarifying information that you may have discovered after your initial recommendation was provided or to respond to a request or question from a customer, as examples.
      • Direct Email: uTest Project and Community Managers place a high level of trust in our TTL community. Occasionally TTLs will be given access to confidential and/or proprietary information including tester email address. TTLs must ensure they treat this confidential information with the upmost sensitivity.
        NOTE: If you are emailing more than one tester at a time you must place email addresses into the BCC field. This will ensure that tester email addresses remain hidden from fellow community members.
  2. Triaging Issues
    If possible, triage reports as they come in (or in short intervals of time) to prevent a pileup of work. Doing so allows the TTL to easily spot duplicates and potentially identify areas of the application that may need further clarification from the customer (in these cases, contact the PM immediately). Additionally, testers will appreciate timely triaging to help direct them where to focus their testing efforts for subsequent testing. There are two levels of permissions for TTLs: Basic and Approval. The Basic level allows the TTL to send an approval recommendation for how an issue should be triaged, while Approval allows the TTL to effectively triage as a customer or PM. Basic-level permissions require the TTL to complete several pre- determined dropdowns that provide more information to the customer and PM – including whether the issue is valid, in-scope, reproducible, and a duplicate of a previously-reported issue (hereby labeled the TTL Recommendations Workflow). Please consider the following items in triaging issues:

    • Triage fairly and consistently: The age-old question is whether to offer testers a 2nd chance when they do not follow instructions, or use a ‘tough love’ approach that will signify the importance of high quality with every single interaction. In an effort to save time and resources, our preference is to offer the ‘tough love’ approach, unless the report is salvageable and could be valuable for the customer. Therefore bugs that violate the customer specified bug templates should be rejected as “Did Not Follow Instructions”, while high-quality bugs and reports that have minor mistakes can be salvaged using “Request More Info” tools (more on that below). Testers can discard their bugs up until the TTL triages their bugs, including sending it to the “Request More Info” state.
    • Bug Templates – Is the report in the proper format (if the customer asked for a specific format)?
      • Confirm that the bug title contains the appropriate information.
      • Confirm that the steps to reproduce are clearly documented as required by the customer or development. Testers should be urged to number their steps to make it easier on the customer to read and reproduce.
      • Confirm Expected Results and Actual Results are documented clearly and provide factual information.
      • Confirm that all appropriate attachments are included as required.
      • Confirm that all the additional environment information is included.
      • Confirm that all other customer specific format requirements are met.
      • Confirm that the bug is classified appropriately.
    • Bug Reproducibility – Is there sufficient evidence supplied in the bug report?
      • i. When possible, TTLs should attempt to reproduce each submitted issue. In some cases the TTL is not able to try to reproduce a defect before triaging it due to device or time constraints. In these instances, the TTL should ensure the tester has provided sufficient proof that it happened during their testing (logs, screenshots, videos, clear and complete steps, etc.).
      • ii. If the TTL is unable to reproduce the issue on the same device/OS as the tester, they may consider asking the tester to verify the issue is still happening to verify the steps written are full and complete. They may have either missed a step or the issue may have corrected itself. Given the bug has adequate information to verify the tester was seeing the issue at the time it was reported, non-reproducibility should not be a reason to reject a tester’s issue before the customer has reviewed it.
      • iii. If there is missing information or more clarification is required from a tester, the TTL can push a bug report to the “Request More Info” status, which shows that the bug report was reviewed but requires a response from the tester.
        Note: Sending reports into the “Request More Info” status automatically adds a note to the Tester Messenger and prevents testers from discarding these bugs. However, it is possible to visually note when a tester has responded through the Platform UI and through email notification as the status of the bug returns to “New”. In addition, this status assists the testers when they see their bug change status. Testers should be reminded to press “Send Requested Info” button. Please note that if the tester does not respond before the cycle closes, then their bug will be automatically rejected.
    • d. Bug Classification – Is the severity and bug type of the issue appropriate?
      • i. If unsure, clarify with the PM about severity and value expectations. For example, what do they consider “Critical” vs. “Low”?
      • ii. If the Severity is not appropriate, send the tester a message asking them to update it, or the TTL can update it themselves if they have the appropriate permissions in the platform. TTLs can also leave a message for the customer’s Test Manager to make the update when they are approving bugs.
      • iii. If the type (functional, UI, technical) is not appropriate, send the tester a message asking them to update it, or the TTL can update it themselves if they have the appropriate permissions in the platform. TTLs can also leave a Customer Message for the customer’s Test Manager to make the update when they are approving bugs.
    • e. Duplicate Bugs – Is the issue a duplicate of a previously-submitted issue?
      • i. If you believe it is, reject the issue as a duplicate. Be sure to cite the bug report # that the rejected issue is a duplicate of. The time stamp for each defect entered into the platform is critical. When deciding which defect is a duplicate and which defect to approve, bugs reported earliest take precedence.
      • ii. If the customer does not want the TTL to do any actual approvals or rejections, the TTL can recommend that the defect be rejected as reason Duplicate and provide the # of the original defect for reference.
    • f. Out of Scope – Does the bug appear to be in scope?
      • i. The scope is determined by the customer and should be clearly defined in the Scope and Instructions provided by the PM.
      • ii. If the bug is not in scope, recommend rejection using “out of scope” as the reason.
      • iii. If the bug is not in scope and the TTL has the ability to approve and reject, then simply reject the bug using the “out of scope” reason.
    • g. Bug Validity – Does the bug appear to be valid?
      • i. Most submitted defect reports will be clearly valid or invalid for one of the reasons above, but some will need further consideration. It may be worthwhile to try to reproduce anything you are in doubt of, to get a clearer idea of what issue is actually being reported.
    • h. Bug Approvals – if the TTL has the ability to approve bugs, they should be very careful that they approve the bug at the proper value. Do not accidentally approve a bug as Exceptional when it should be Very or Somewhat valuable.
    • i. Customer Approvals – How to deal with the reported issue that is a valid defect from a tester standpoint but not a valid issue from the customer’s perspective?
      • i. In this scenario, there are two actions that the TTL can take, depending on the individual circumstances:
        • i) Reject (or recommend rejection) as Working as Intended/Designed. This reason does not negatively impact the tester’s rating but they also will not get paid for the defect.
        • ii) If applicable, Approve (or recommend approval of) the defect but leave a Customer Note that indicates that it should not be imported into the customer’s tracking system. Defect value is normally set to “Somewhat valuable” for these.
      • ii. In the case where the test application is either overly complicated or the customer does not have a complete list of known issues/working as designed, it’s more fair to the testers to approve the defect so they are paid for the time they put in reporting it. Too many “Working as designed” rejections cause testers to shy away from testing and can cause participation rates to drop.
      • iii. If the average tester would consider the issue as working as designed, then a rejection is the appropriate strategy. Use your best judgment when making this decision.
    • j. Bug Export – If the bug is valid, does it need to be entered into the customer’s bug tracking system?
      • i. If so, and the TTL will be handling the bug tracking system import, there are two ways this can be done.
        • i) If the account is set up for it, you can use the “Export to Tracking” button on the bug report in the platform to automatically move the report into the customer’s system. This functionality is customer facing. If a TTL is given an account to access customer site they must be aware that they should not use that account to triage bugs. TTLs should only use their tester/TTL account to triage issues. Under no circumstances should TTLs approve their own bugs.
        • ii) If the account is not set up for it, you can manually copy the bug report from the uTest JIRA system to the customer’s bug tracking system. The most efficient way to do this is to click the Copy link at the top of the defect screen in the platform and paste it into WordPad, then copy the sections you need into the corresponding sections in the customer’s tracking tool (JIRA/Team Track/etc.).
  3. Triaging Test Cases
    Depending on the test cycle and PM, TTLs may be responsible for approving or rejecting Test Cases. TTLs should always open each Test Case submission and check for all requested materials before choosing to Approve or Reject. Please consider the following points in triaging test cases:

    • a. Did the tester include all requested documents or information with the Pass/Fail verdict?
      • i. If attachments were requested, confirm that there actually is an attachment. Then be sure to open them and verify they are in the correct format and contain the necessary information. If multiple submissions were made by a single tester, compare these documents across the various completed test cases to verify they are original and not simply duplicates with different titles and dates.
      • ii. If the tester was asked to do exploratory testing, make sure the exploratory notes are filled out completely and cover a sufficient amount of testing time dependent on the expectation set in the overview or based on the payout (i.e. a $10 test case will probably have a lower expectation of work than a $30 test case).
    • b. Were the testers required to sign up (usually via an external document) for a limited number of test cases? If so, check to be sure the tester’s submission you are approving is slotted for a test case and that the device/browser/etc. details match.
      • i. If the testers were required to sign up, were they instructed to only fill a specific number of slots? Keep an eye on testers who try to reserve more than their share of test case slots.
  4. Bug Fix Verification
    If the TTL is working with a customer who either has bug tracking integration with our platform or for whom you are manually transferring our defects, you may also be responsible for identifying the need for (and possibly kicking off) Bug Fix Verification (BFV) cycles. Clarify with the PM who will be responsible for BFV management before the test cycle starts. Keep an eye on the fix statuses of defects that have been moved from the uTest platform to the customer’s bug tracking system and either manage the BFV cycles in a timely fashion or notify the PM which issues need attention. If a bug is verified fixed by the testers via BFV, be sure to update the status to the corresponding bug in the customer’s tracking system. If the testers find that that the issue is still recurring, report this status in the other system and continue to monitor it there, kicking off another round of verification testing when warranted. Again, be sure to clarify with the PM who will be responsible for BFV management before the test cycle starts.
  5. Disputed Bugs
    If a bug is disputed by the tester, the TTL should review the comments and then speak to tester on Tester Messenger to notify the tester that their bug is being reviewed. If the TTL has control of rejections and approvals, they can determine a final status. If rejected, the bug changes to Under Review for the PM to make a final determination. If the TTL made a mistake in the initial triage of the bug, they can recommend to the PM that the dispute be resolved and the bug approved.
  6. TTL Testing on Test Cycles
    TTLs should discuss whether they can test on the same test cycle that they are TTLing with the PM for that test cycle before launch. If the PM agrees that the TTL can test on the test cycle, then the TTL should notify the testers through the welcome message in chat to reduce confusion for the testers.

Expectations and Escalations

The following list contains the expectations and escalation paths specific to all TTLs. The tester Code of Conduct applies in all cases for all testers and TTLs.

  1. Expectations: TTLs are expected to…
    • Respond as quickly as possible to questions in chat for the test cycle. The availability schedule for the TTL and the PM should be listed in the Scope and Instructions. While testers typically expect chat responses within an hour of submission, TTLs should attempt to respond with 2-4 hours during normal business hours and within 12 hours during the nights and weekends.
    • Closely monitor test cycles directly after it launches for questions in chat and issues with bug reports.
    • Monitor chat for appropriate tester behavior and escalate any issues that are observed.
    • Help reduce the noise created by testers and reduce the number of information requests that PM’s and TM’s must address by gaining a strong understanding of the application and the scope of the test cycle and taking the lead role in answering questions in chat or in the defects.
    • Triage defects and test cases during the test cycle. Feedback (including approvals, rejections, or requests for more information) to testers during the test cycle can result in more detailed information in the bug reports and higher quality bug reports and test cases available to the customer. This increase in response to testers can assist in improving the overall quality for subsequent bugs in the current test cycles and future test cycles.
    • Provide feedback to testers based on their knowledge of the test cycle, the application under test, the customer, the devices under test, and the expectations listed by the PM prior to test cycle launch. It is understood that this information may change test cycle to test cycle or even within a single test cycle.
    • TTLs should provide feedback to testers when expectations are not met in defects, test cases, and reviews. They may also provide feedback for exceptional work.
    • Be flexible, very flexible.
      NOTE: If a TTL is unable to commit to the expectations of a particular test cycle they must give the Project Manager sufficient notice so that the role can be successfully backfilled. Sufficient notice will vary by test cycle, at the very least a TTL must give 48 hour notice that they will be unable to perform the work required. However, if there is a longer on-ramping time associated with a cycle, then the TTL should inform the PM two weeks prior to exiting the role of TTL.
  2. Escalations:
    1. For any issues dealing with the test cycle, the TTL escalates to the primary Project Manager first. If there’s not a response within 24 hours for normal issues or 12 hours for urgent issues, then the TTL should send an email to PM.Escalate@utest.com with a subject line in the following format – [TTL Escalation for Customer Name and Test Cycle ID#] and details on the issue in the body of the message. A TTL should only directly contact the customer if permission has been granted by the Project Manager, either at the project or test cycle level.
      Example 1: Testers record questions in chat about the fact that there is no review tab active for the test cycle, though the scope refers to customer reviews when testing is completed.
      Example 2: For an e-commerce site, testers are asking whether the entire shopping cart function is out of scope or only the final purchase step.
    2. Any issues with testers behavior, Code of Conduct infractions, or otherwise should be escalated to CM with notification to the PM.
      Example 1: Tester discusses test case pay out rates in chat.
      Example 2: Tester disputes rejected bugs multiple times.
      Example 3: Tester misuses slotting spreadsheets.
    3. As mentioned above, if a TTL is unable to commit to the expectations of a particular test cycle they must give the Project Manager sufficient notice so that the role can be successfully backfilled. Sufficient notice will vary by test cycle, at the very least a TTL must give 48 hour notice that they will be unable to perform the work required. However, if there is a longer on- ramping time associated with a cycle, then the TTL should inform the PM two weeks prior to exiting the role of TTL.

TTL Bill of Rights

  • Receive notification of test cycle assignments prior to launch of the test cycle
  • Receive anticipated scope of work prior to TTL accepting each TTL assignment
  • Receive information about anticipated payout prior to TTL accepting each TTL assignment
  • Receive notification from the PM prior to the launch of the next test cycle if PM decides to swap TTLs for any reason
  • Receive a response from the PM within 24 hours for normal issues or 12 hours for urgent issues during the following timeframe: 8am – 8pm US Eastern time. If you don’t hear back, you may email to PM.Escalate@utest.com with the customer name and test cycle ID in the subject line
  • Expectation that testers will contact TTLs through the platform only, not through any other method like Skype, IM, Google+, etc.

uTest Business Model

For TTLs who have not experienced uTest for a long time, here is a brief outline of uTest’s business model. This knowledge should put things into perspective when you have outstanding questions. However, the Community Management Team is always available to address any open questions about the uTest business model and how it pertains to any TTL assignments – past, present, or future.

The uTest business model is built on these Customer Expectations – that uTest:

  • Provides consultation for test strategy, test cycle configuration, and test cycle optimization to achieve high-quality results
  • Provides various encapsulated industry and domain-level knowledge and experience
  • Provides testing expertise and diverse testing coverage
  • Provides high value defect reports
  • Responds quickly to customer and testers alike
  • Assures a low level of noise during each test cycle

Basic Business Concept:

  • Larger customers generally engage uTest for longer timeframes to ensure continuity of testing coverage and expertise. In general, Project Managers that cover larger customers have fewer total customers, as this allows for deeper focus on customers who generally have greater needs.
  • Smaller customers may engage uTest for a variety of time periods, whether it’s several test cycles or annual contracts. In general, Project Managers that cover smaller customers will have more customers, which generally results in slower response times.

Key Message for TTLs: No matter the size of the customer or the size of the test cycle, every customer should be delighted with the test cycle experience from activation to close.

TTLs and SDLC

A basic understanding of the Software Development Lifecycle (SDLC) is very useful for every TTL. The more confidently that a TTL understands the different stages of the SDLC the better that the TTL can prepare and set appropriate expectations for the testers on the test cycle. In general, applications that are tested earlier in the development cycle (like an alpha release) have code that is relatively untested, and testers can expect to find more bugs. On the other hand, applications that are being tested later in the development cycle will have more stable code with fewer bugs that are more difficult to determine and more critical to the live release of the software. Please refer to the Wikipedia definition of the SDLC to begin your research on this topic.

Conclusion

The term TTL is a uTest-derived acronym that stands for ‘Test Team Lead.’ In a Test Cycle, a TTL is the bridge that connects Project Management, Testers, Customers, and Community Management and assists in increasing the uTest value to the customer and the testers.

TTLs have 3 primary tasks in a test cycle:

  • Platform communications
  • Triaging Bugs
  • Triaging Test Cases

Be sure to confidently understand the standards and expectations listed in this handbook before engaging in any test team leadership activities. Because it cannot be repeated enough, the goal for the TTL is to increase the value that uTest brings to the customer by maximizing the output and minimizing the noise level. As a TTL, we are placing a great deal of trust in your technical and leadership abilities to help manage each test cycle. Therefore, please do not hesitate to reach out to the CM Team if you do not believe we are providing appropriate training or expectations for all parties involved in order for you to become successful and for our customers to extract the highest value from uTest. Lastly, feel free to reach out to the uTest Community Management Team if you have any questions or concerns with current or recent TTL assignments.

Recommended Next Steps

Once you feel comfortable with the information contained within the TTL Handbook you may request access to the TTL Entry Level Exam by emailing TTL@utest.com.

  • Participate in on-line training sessions as they are available.
  • Further training may be required:
    • Rules about exporting bugs:
    • CSV
    • Bug tracking systems
  • Communications with PMs
  • Communications with CM

Addendum

Chat Examples:

Example: — New Chat Topic – Welcome message:

When the test cycle launches, the TTL should introduce themselves as the Test Team Lead for this test cycle in the Chat Topics. This notification establishes a point of contact testers can reach out to with their questions and observations. A generic welcome message is provided below. Amend as appropriate:

Welcome to this<Project name> cycle. I will be your Test Team Lead on this project. At any time, please let me know if you have any questions or concerns. I will try and find an answer for you as soon as possible.

Please ensure that you read the scope and instructions carefully. Check the Out of Scope section and known issues list before starting your testing effort.

In addition, make note of other bugs raised during the cycle to assure that you do not raise duplicate bugs.

When raising issues, please ensure that the bug titles and descriptions are of the highest quality. They should be descriptive and clear. The bug title in particular should be descriptive enough to explain the area and issue so it is easy to spot when going through the bug list.

Please be sure to include all mandatory information for the bug as required.

Thanks a lot and happy testing

<TTL’s Name>

2014 Year in Review

I’d love to write a witty post recapping the last year, but right now I just don’t have the time. So a brief summary will have to do for now.

Job Change

In February, I decided to leave Brock Solutions and work for uTest as a full-time consultant. I spent the next 8 months working for SalonCentric (a L’Oréal devision) as a business/test manager as they built and launched their new eCommerce website. That was wonderful time. I make some fantastic friends and learned a lot about testing eCommerce, particularly on the Demandware platform.

I was promoted to an Enterprise Test Team Lead which is essentially a contracted Test Project Manager. I was assigned several customers and project all of which were delivered successfully and received praise and accolades.

uTest

University – I wrote or co-wrote several courses for the uTest University

Forums – I stayed quite active on the forums as usual. I’m almost at 2,000 posts.

  • I started a really fun series titled Analyze This where I posted a testing “truth” and we debated it. Although technically I started this in 2013, the majority of the posts happened in 2014. So far I’ve started 13 topics, all of which earned at least 2,000 views and several are approaching 10,000!
  • I also made a very controversial proposal to change the uTest rating system to eliminate rating hits. There’s a lot at stake so I understand uTest’s hesitance to move on it, but I still think this is the right thing to do.

Blog – I wrote a few posts for the uTest blog, two of which were selected for The Best of 2014: Top Posts from the uTest Blog. “Certified and Proud?” was one of uTest’s most popular posts of the year, earning the largest number of comments and social shares. Even a few A-list testers read and talked about it!

A.C.E – uTest started a new initiative to provide testers with organized, mentor-based training. I helped put together the pilot program and was one of the mentors for it. A large portion of the materials we used were things that I’ve written for uTest over the years. I’m now working on converting all of my previous articles and papers into more structured courses as the A.C.E. program continues to evolve and improve.

Awards – uTest created a Hall of Fame website and introduced quarterly awards based on votes from the community. In the two quarters that this as been going on, I’ve won 3 awards

  • Outstanding Forums Contributors
  • Outstanding Blogger
  • Outstanding uTest University Instructor

AST-BBST

In September, I successfully completed the Foundations Course. That was quite an experience. I hope to find some time to write about it one day.

In November, I passed the Instructors course and will be an assistant instructor for the first Foundations class of 2015.

uTest’s Testers of the Quarter (2014 Q4)

badgeTesterOfQuarteruTest recently announced the Testers of the Quarter winners for 2014 Q4.

This quarter I took home the hardware for:
Outstanding University Instructor because of the Exploratory Testing course I co-authored with Allyson Burk.

uTest University Webinar – Live Exploratory Testing

To go along with my course on Exploratory Testing, I had the opportunity to do a live webinar demonstrating how I do exploratory testing. The community management team did some great marketing and over 300 people signed up! Unfortunately, we found out at the start of the webinar that the uTest GoToMeeting license only supported 100 participants so a lot of folks were not able to attend. It was quite a humbling experience to have so many people interested in watching how I test.

As you can see from the video, the app I chose was quite buggy. That coupled with my poor internet connection presented some interesting challenges. It was difficult to maintain my train of thought and to investigate a single issue when other issues kept popping up, but I think it gave a good representation of what exploratory testing can be like sometimes.

Looking back at the video I need to improve in the following areas:

  • Speak slower and clearer – I was trying to articulate my every thought and this lead me to speak in quick, incomplete sentences. It might have been beneficial in showing how choppy my thought process can be, but I think it might also made it difficult to follow what I was thinking.
  • Repeat the question – During the Q&A portion, I didn’t do a good job of stating the question I was answering. I read the question and kinda mumbled it before jumping into the answer. I did OK on some questions, but I need to remember to do a better job of this next time.

http://university.utest.com/recorded-webinar-exploratory-testing-lucas-dargis/

uTest University Course – What is Exploratory Testing?

I recently authored a course with my good friend Allyson Burk for the uTest University on Exploratory testing. This is a heavily viewed course because it’s part of the “Getting Started at uTest” course track which most new uTesters work through. Since this course is a foundational piece of new uTester’s development, I spent a large amount of time researching and editing to make this as accurate and consumable as possible. So far it’s been well-received earning a 5-star average rating – Not too shabby.

https://university.utest.com/what-is-exploratory-testing/

It even got front-page billing on the University home page:
University Home Page - ET

 

Below is the content of the course:

INTRODUCTION

Today’s development world is much different than it was 10 or even 5 years ago. Product development is fast-paced and must meet the high expectations of users. As development practices have changed, so too have approaches to testing. Many testers are finding that Exploratory Testing (ET) is an effective way to test in these circumstances. The adoption and use of ET has rapidly grown, to the point where it is arguably the most popular testing approach used today.

In this course we’ll start by learning what ET is and how it differs from scripted testing (ST). Then we’ll look at why you should use ET and finally, we’ll wrap up by showing you how to get started. So let’s make like Magellan and start exploring!

THE ‘TRADITIONAL’ APPROACH TO TESTING

Before we look at ET, it might help if we first talk about a different, more traditional approach to testing so we can use that as a reference point to make some comparisons.

Scripted Testing (ST) is a two-step approach to testing. First the tests are written; they are planned, designed and documented. Second, the tests are executed. These two activities are done independently of each other and in many cases, the person who writes the tests is different than the person who executes them.

Generally, the tester executing the tests has some knowledge of the product, or the tests include the information needed to execute them. This is important because without that knowledge or information, the tester might not be able to execute the test or interpret its results.

WHAT IS EXPLORATORY TESTING?

Now lets compare that with ET which is simultaneous learning, test design and test execution. In other words, the tester is designing his or her tests and executing them at the same time. As an exploratory tester, your next action (the next test) is influenced by your previous actions, your observations of the product’s behavior, and your own thought process.

ET also assumes that a significant portion of the testing will be spent learning about the product. As you explore, you become more aware of how the product functions and its expected behavior. You can use that knowledge to design new and better tests. It also helps improve the analysis of the test’s results.

ET table

It is important to make the distinction between ET and other types of unscripted testing because some testers mistakenly believe that all unscripted testing is simply poking the product randomly to see what happens. Performing a series of random actions is called monkey testing and in some cases it may be a valid approach, however this is quite different from ET. With ET actions are the opposite of random — they are deliberate, driven by human thought and reasoning. Your approach is continually refined as new information is gathered and analyzed.

When an explorer goes to an uncharted region of the world, they spend months preparing. They go with a goal in mind and they rely on their abilities to adapt to changing situations. Similarly, an exploratory tester must prepare. They too have a goal and the skills needed to adjust their course. It’s true that monkey testing may occasionally find useful information, but it’s found unexpectedly. It’s the difference between discovery and exploration; luck versus skill.

WHY DO EXPLORATORY TESTING?

One of the most defining qualities of humans is our ability to think. It’s what makes us unique. It allows us to analyze a situation and make decisions, or come up with new ideas, or find new solutions to a problem. We have the capability to learn and continuously improve.

Henry Ford once said “If you always do what you’ve always done, you’ll always get what you’ve always got.” Executing scripted tests over and over will generally produce the same results. Exploratory testers use their cognitive abilities to continually improve the value of their work. They explore and adapt, they learn and adjust. ET is designed to make the most of our intellectual abilities.

ET also takes advantage of the differences between testers. Each tester’s previous experience, skills, and thought process (among other things) causes him or her to view thing in a unique way. Different testers may come up with different ways to test the same function. It may be more beneficial to have three testers test the same function in three unique ways than it is to have all three test it in exactly same way.

In many cases we need to use ET because there is no other alternative. For example, consider a situation where the tester isn’t familiar with the product under test and scripted tests are not available. In this case, it’s up to the tester to study the product, design and execute their tests. As we’ve already seen, this is the very definition of ET.

It can take a significant time to read, comprehend, and execute each step in a test case. This is especially true if you don’t already know the product, or the TC uses language or terms you’re not familiar with. When a tester is tasked to find bugs quickly, they need to be searching for bugs, not reading test cases. They need the freedom to follow promising leads, not the constraints of predefined instructions.

HOW DO I GET STARTED?

If there is one thing all new testers (including new exploratory testers) should do, it’s to start by thinking about the product in general terms; try to see the big picture. Instead of initially focusing on one specific thing, first try to understand the context in which you are working.

Some questions to consider are:

  • Is this a product in development or is it already in production?
  • What is the purpose of the product?
  • Who are the users and how are they going to use it?

Jumping right in and banging on things might produce a bug or too, but if you hope to get the most out of ET, initial preparation and understanding your context is vital.

Now lets see how ET might look in practice. Imagine you’re a brand new tester, your boss comes to you and on your first day and says, “Here you go, this is the latest version of our app. Please begin testing and report any bugs you find.” There are no test cases and no documentation. What do you do? An exploratory tester would do something like this:

1. Get a notebook (or a digital word processor) to take notes as you go.

2. Explore the app as if you just downloaded it and want to use it yourself. If it is not an app you would typically use, then imagine you are the target market for the app.

Take a moment to really get in the mindset of a typical user. Some questions you can ask yourself are what is the goal of this app? Who would benefit from that? How do they benefit?

Let’s say this is an app to show up-to-date stock market information.

Goal of the app: Having stock market data at your fingertips.

Who benefits: Someone who is financially savvy or wants to be and has available income to be investing or has interest in other people’s investments.

How do they benefit: They benefit by the data being timely, accurate, easily accessible and displayed in a way that they can understand quickly.

Don’t worry about finding any bugs right now. You may stumble on them, but this is really just getting used to the app. Jot down anything you find that you want to explore further later.

3. Once you get a feel for the app start going back to the areas that interested you and you thought might be a place of vulnerability in the app. This knowledge about vulnerability is going to come with experience. Don’t worry if you don’t have any experience yet because you are about to get some!

4. One by one, work through each area you’ve earlier identified, exploring every function in that area. Think of what a real user might do. Come up with with some use cases or scenarios and execute those. Then think of variations and execute those. Use the results of your tests to help you come up with new ideas.

5. Focus on one bug at a time, but always be on the lookout for hints of other bugs or suspicious areas. In your notebook, quickly make a note of these areas and how to get back to them. This way you can come back and explore each one later. You could very well end up with 4 or 5 bugs just from investigating the initial bug.

6. Once you’ve exhausted that area or function of the app, move on to your next point of interest. As you repeat this process, remember what you’ve learned so far and use that information to influence your tests.

As you can see from this narrative, you are simultaneously learning, designing tests, and executing the tests. These are the core pieces of ET. Understand this and you’re on your way to becoming an exploratory tester.

CONCLUSION

Different testing needs call for different testing approaches and there are many situations where ET can prove most beneficial. ET is the inverse of scripted testing because it relies on human intellect as opposed to simply following instructions. ET is a process of continual refinement and improvement where testers adapt to situations and the information they’ve gathered. Now that you’ve been introduced to ET, our hope is that will continue to explore exploratory testing and that you can use these skills to provide the most value possible.

ADDITIONAL READING

uTest blog post – Three Things Testers Can Learn From Wine Experts

Originally posted on the uTest blog: http://blog.utest.com/2014/08/22/three-things-testers-can-learn-from-wine-experts/

Also posted here: http://blogs.yourzephyr.com/?p=3987

When I’m not testing, one of my favorite hobbies is alcohol. Wait…that didn’t come out right. What I meant was my hobby is learning about winSommelier_e_Tastevine, beer and sprits. Yeah, that sounds better.

While I do love a cold beer in the summer, a single-malt scotch when I’m feeling sophisticated, or an 1855 classified Bordeaux on special occasions, I think I spend more time studying booze than I do drinking it. I really enjoy learning about the various Appellation d’origine contrôlée (AOCs) in France, and the differences between a Pinot Noir from California and one from Burgundy. I sound pretty smart, huh?

As any cultured, refined wine connoisseur such as myself knows, the true masters of the bottle are called sommeliers. These fine folks are highly trained adult beverage experts who often work in fancy, fine-dining restaurants, setting the wine list, caring for the cellar and working with customers to help them select the perfect wine.

So what could a tester possibly learn from a someone obsessed with booze? Good question! I have three answers.

Be passionate

I have yet to find people who are more passionate about what they do than Master Sommeliers. Need proof? Watch the movie Somm (available on Netflix). The tremendous amount of dedication and effort these people pour (wink, wink) into their work is simply astounding.

A sommelier must be constantly learning and exploring. Each year, a new vintage of every wine is created. That means thousands of new wines are added to the multitude that already exist…and a sommelier is expected to be familiar with with all of them. And you thought the IT world was constantly changing!

There will always be a new product to test, a new approach to learn, a new idea to debate. Testers who are passionate about testing are excited about these new developments as they are opportunities to grow and improve.

Be a servant

From the Demeanor of the Professional Sommelier:

It is important for sommeliers to put themselves in the role of a server; no job or task on the floor is beneath the role of a sommelier; he or she does whatever needs to be done in the moment to take care of the guest.

Sommeliers are at the top of the service food chain. They are highly trained, knowledgeable, and professional people, yet they are also the most humble servants. They realize that their qualities alone are worthless. They must be used in the service of a customer for their value to be realized.

Testers too need to remember that we don’t get paid because we think we’re great testers. We get paid because the person who is paying us values our work. Put your client’s perception of value and quality ahead of your own.

Be an expert

A sommelier would never walk up to your table and say, “I recommend this bottle I found in the back.” They are the ultimate experts in wine selection and food pairing. He or she asks questions: What do you like? What have you had before? What is your budget? They learn about the customer and use that information to help them find exactly the right bottle.

Likewise, a tester should be knowledgeable in the testing field. A good tester doesn’t just randomly go banging on things — they too take a more thoughtful approach. What areas are the most error prone? What parts of the product are the most important? What information would be most useful at this stage in the product’s life cycle? They learn about the product and the circumstances under which they are testing to ensure they provide the most value possible.

Take pride in your work. Understand that testing is not a low-skilled job; it is a highly cognitive profession with demands on your professionalism, communication skills, and attention to detail. It takes a lot of effort, study and experience to become an expert (or so I’m told), but that should be the goal of every tester.

Decision Review: Pursue my MBA

My first Key Decision was to Pursue my MBA. Let’s see how that decision worked out shall we?

After only one semester it became quite clear that pursing my MBA was a poor decision. Hmm… so far I’m 0 for 1. You might think I quit before I even gave it a chance, but I struggled just to finish out the semester. After a lot of reflection on what exactly went wrong, I was able to narrow it down to three things. The impact on my family, the educational aspect was disappointing, and the opportunity costs were too high.

Time with family

Going in, I knew that class time would reduce the amount of time with my family, but that didn’t fully sink in until it became real. A 3 hour class, sandwiched between a 45 minute commute meant that two days a week, I wouldn’t get to see my wife or kids. How could I possibly be so cruel as to deny my family the pleasure of my company? My absence did put a lot of additional burden on my wife and the kids missed wrestling with me, but mostly I’m just selfish; I missed them too much.

Disappointing

After my first class I was super excited. The instructor was fun and engaging. He asked open ended questions and joined myself and others in debate and discussion. Unfortunately that didn’t last long. As we approached our first exam it became clear that grad school is merely a continuation of undergrad. You’re still graded and evaluated on what you know, not on how you think, reason, or your ability to learn and execute. Multiple-choice Scantron tests…SERIOUSLY!? The entire dynamic of the class changed. Boring lectures attempting to “teach” the “right” answers to unimportant questions.

I was really looking forward to learning from my fellow students, from their experiences in industries and business areas new to me. Sadly, most only cared about their grade. “Is that going to be on the test” was the most common question asked. The lack of interest in true self-improvement and overall “quality” of the students admitted to the program was disheartening.

To be fair, I only took one class at one university, but as UNCG is a highly ranked university, I have to suspect that my experience isn’t all that unique.

Opportunity costs

Similar to how my time focusing on school reduced my family time, it also took the place of other career-related opportunities, specifically uTest. The additional 20 hours a week meant that I had to completely abandon uTest. My good friend Rex helped me realize what an expensive trade-off that really was. I had spent two years building my reputation as a tester and done so quite successfully. I was invited to best projects, I was able to participate in various company initiatives (like the TTL training and evaluation program) and I had a large, visible presence in the community. I was sacrificing a opportunity that provided me continuous growth opportunities, respect, and enjoyment for the chance to become one of the select 100,000 MBAs that graduate each year.

What I learned

Structured, standardized learning no longer appeals to me. I learn more from conversations over a few beers than I do listening to a lecture. The ability to execute is more important than the ability to memorize business trivia. Doing something just because it’s uncomfortable is not a good reason. I thought I was being courageous by stepping out of my comfort zone, but know I see I was just being an idiot.

In the end I realized that the expectations I listed in my decision post can be fulfilled by continuing to develop my testing career. I was already working towards all of them and making great progress. At this point in my life and career, the benefits of an MBA program didn’t outweigh the costs. Maybe someday I’ll regret not having those three letters after my name, but today is not that day.

Webinar – Finding bugs in mobile devices

I was able to join Kayla Cox and Todd Smith for a uTest webinar to talk about testing mobile devices and how to find high-value bugs.

Since my microphone was terrible (and I might have been mumbling a little ) here is a summary of the points I made in our discussion.

Crashes
Understand that not all crashes are valuable.
Out of memory crash may be due to other apps using up 90% of your memory and the app you are testing just pushed you over the limit. The best way to know for sure, is to have a clean test bed. Restart your phone after you install a new app, and make sure no other apps are running in the background.

When you do get a memory related crash, use a memory management app to help you see where your memory usage spikes. Being able to identify a reproducible memory crash is usually a high-value bug

Connection Issues

  • Kill your connection while data is being transferred
  • Unplug your wi-fi router/modem
  • Turn on airplane mode
  • Turn of wi-fi on your device
  • Turn off cellular data on your device
  • Find places near you that have low or no signal and test there

Interaction with native and popular apps

  • Share something via email with no email set up
  • Log in using Facebook account with/without the Facebook app installed
  • Interrupt testing with phone calls, text messages, FaceTime calls etc
  • If the app changes the phone settings, make sure it does it correctly. Change it back manually in settings and see how the app responds

Investigation and Documentation
There are many topics on how to write good bug reports but there are a few points worth reiterating

  • Provide exact reproduction steps
  • Do root cause analysis – don’t report symptoms. I once saw 3 testers reported 3 different symptoms of the same bug. On the surface they all looked like different bugs, but a little analysis showed they were all caused by the same step they all overlooked. 

Checking vs. Testing – The Problem with Requirement Documents

The prevalent idea that testers are dependent on a requirements document to do their job is a dangerous one. Requirements are not always needed to test. In fact, in many situations, they may actually reduce a tester’s effectiveness.

The process of deriving tests directly from the requirements has several names. The ISTQB uses the term “specification-based testing”, sometimes it’s referred to as “Happy Path” testing, but I think the most appropriate name is “checking”. Michael Bolton wrote a well-known post about this topic (http://www.developsense.com/blog/2009/08/testing-vs-checking/). Checking is confirming that what we believe is actually true. Products are built in accordance with the requirements, so the requirements are what we believe to be true. When we verify that our product meets the requirements, we are “checking” the product. When a tester relies on a requirement document to test, he isn’t testing, he’s checking.

When we test, we are exploring, investigating and learning. Our actions are influenced by new questions and ideas that haven’t yet been explored. The use of requirement documents while testing can cause problems because it can give a false sense of test completeness, it can steer testers in the wrong direction, and it can reduce the independent thinking of the tester.

Gives a false sense of test completeness

If we have verified that our product meets all of the requirements, does that mean that the product has been well tested? True, you have verified that the product behaves the way we expect it to (in specific situations) but you still don’t know how the product will behave in situations not specified in the requirements.

“That wasn’t in the requirements” I’ve heard testers use this excuse many times and it drives me crazy. As a tester it is your job to investigate the product to learn about areas and behaviors outside of the requirements. A product is much more than its conformance to the requirements. It’s up to you to cover that gap between what is expected of a product and what actually is.

Checking that a product meets the requirements is necessary of course, but checking alone does not indicate test completeness.

Steers testers in the wrong direction

When you first start testing a new product or feature, what should be the first thing you test? Some might say you should verify the requirements. I challenge that view. I think the requirements should be one of the last things you test. In my opinion, a developer who writes code that didn’t meet the requirements failed to do his job. Before the test team sees the product, the developer has already spent hours working and verifying that his work is correct. Although possible, a strong developer rarely produces work that doesn’t meet the requirements. With that in mind, there is little value in retesting his work, especially when you consider the other aspects of the product that haven’t been tested yet.

Requirement documents can stifle the creativity of exploratory testing. When testers have a requirements document in front of them, they may be more likely to verify the requirements first and focus on areas where the likelihood of learning new information is the lowest. Instead, they should focus their efforts on new tests and unexplored areas where the opportunity for learning is the highest. When testing without requirements, you eliminate its influence on your testing decisions; you have to rely on your own abilities, your knowledge, and your curiosity. You test.

All testers have heard the old saying “Trust but verify”. I’m not saying that we shouldn’t verify developer’s work, only that there’s more value in performing a test for the first time, then performing the same test multiple times. Focus on the activities that produce the most value first.

Reduces independence

The idea of “independence” (See ISTQB Foundation) refers to how close a person is to a product. A developer who wrote the code would have the least amount of independence, while a person in a different company who has never seen the product would have the most independence. Independence can often be a great quality for testers. Testers with no preconceived notions of how the product is supposed to work are able to view the product with more objectivity.

Consider the common situation where a product was built according to incorrect requirements. The tester was able to verify that the product met the requirements, so was there an issue? In this case the requirements served as a false crutch to the tester. Just because the product met the requirements doesn’t mean it was working correctly. A tester with no knowledge of the requirements would have been better positioned to identify any errors because he wouldn’t have been comparing it to an incorrect “truth”.

We have now seen some reasons why testers should avoid relying on requirement documents. While they may be a necessity for different “checking” activities, testers who wish to provide value thorough “testing” must understand that their value is best realized when they test without requirements.

I’m uTest’s 2012 Mentor of the Year!

For the past 3 years, uTest recognizes uTesters who have consistently gone above and beyond their call of duty. uTest recently announced their selections for the 2012 testers of the year and I was selected as the 2012 Mentor of the Year!

Wow! What a thrill!

As I’ve mentioned many times, uTest provides us testers with many opportunities to grow and develop our testing skills. We are constantly exposed to new products, devices, and customers. The uTest forum always keeps us up to date on the latest testing trends and hot debate topics. But uTest offers us more than opportunities to learn; uTest also provides a platform for us to teach and mentor.

My greatest thrill comes when uTesters comment on how one of my posts helped or inspired them. It’s the motivation behind everything I write. It’s a privilege to be able to influence new uTesers as they evolve into highly-skilled and respected testers.

uTest has assembled a community of testers ready to learn, but that need must be met by those willing to teach. Every tester has knowledge they’ve gained through study and experience. No matter how simple it may seem, that information is valuable. If you’re brave enough to share what you have learned, you’ll experience the amazing feeling of knowing you are positively impacting your community and industry.

I am truly honored to receive this award and I want to extend my sincere thanks to the uTest team and the uTester community.

If you care to read any of my “uMentor” posts, they are all located here.