Key Decision: Pursue my MBA

Decision

The key decision I made is to pursue my MBA degree. I’ve been wrestling with this decision for several years now. I’ve always enjoyed learning and studying but it just seems like there are so many reasons not to do this. I have two children and a wonderful wife that I want to spend time with, I just started a new job, I coach hockey, I’m heavily involved with uTest, etc.

There are also philosophical reasons pulling me away from the MBA. Is an MBA really the best way for me to learn? I’m mostly interested in testing so why not focus on testing courses from AST or personal mentoring and training from some of testings leaders. I should also quote Will Hunting :)

…you dropped a hundred and fifty grand on a f@#$% education you coulda’ got for a dollar fifty in late charges at the Public Library.

So why did I make this key decision? Well… because it wasn’t an easy, comfortable decision. Whenever I find an uncomfortable situation, I try to force myself through it. I don’t know if this is the right decision but I know I’m a bit scared so I know I will learn plenty whether my MBA related goals are met or not. Also, the financial risk is tolerable at UNCG.

Expectations

  • Develop a more complete understanding of business – I know plenty about IT and testing, but other aspects such as finance and marketing are mysteries to me
  • Gain experience working with people from different business areas and different cultural backgrounds
  • Influence the education and growth of others
  • Learn new ideas and new ways of thinking
  • More challenging and rewarding career options

Strength and weakness exercise

“What are your greatest strengths?” “What are your greatest weaknesses?” Two of the most common and dreaded interview questions. Most people think they know the answers to these questions, but I’m pretty sure most people are wrong; especially when it comes to their strengths. The things I’m not good at are fairly obvious, but discovering what I’m truly good has proven to be more difficult. I think this is because what I think are my strengths are what I want to be my strengths.

I’ve found that unbiased self-reflection is a difficult thing to do well, so in attempt to better identify my strengths and weaknesses, I’m going to try to document key decisions I make and what I want and expect to happen as a result of those decision. A year or so later the plan is to revisit these decisions and see if I was able to achieve the desired outcome.

I unknowingly started this exercise in March of 2012 when I decided to join uTest. I documented the reasons for making that decision as well as my goals and now, 18 months later what I expected to happen – develop a reputation as an expert tester – is starting to develop (at least within the uTest community).

I’m excited to see where this exercise takes me. To kick it off, my first key decision is to…. start keeping track of my key decisions :)

My First Testing Experience – Part 2

Continued from Part 1

I can vividly remember lying on my couch in my living room staring at the ceiling, my stomach in knots. I felt this huge weight on top of me making it hard to breath. I was pretty close to panicking. “How can I possibly do this? I have no idea what I’m doing. Why did I agree to this project? Do they have any idea what they’re asking? This is so unfair. This is CRAZY!” These were the thoughts racing through my mind. Before I could think one through, another would jump on the pile. That feeling of hopelessness, of being completely overwhelmed was my first and worst moment as a software tester.

I’d like to think the PM understood the magnitude of the project, that somehow she saw this amazing potential inside me just waiting for an opportunity to prove itself. But I knew she didn’t really understand what she was asking. In her mind this was a small task that any mid-level developer should be able to do. After all, it’s just testing, how hard can that be? That just added to the pressure. I was dealing with unrealistic expectations about an undefined project that required an under-appreciated amount of skill and effort.

After breathing into a paper bag for 10 minutes or so, I calmed down. I reminded myself the best way to climb a mountain is one step at a time. I started by making a spreadsheet documenting all the different migration tools we had built. I identified the base data type, the destination data type and a description of the transformation needed. One by one I went down the list, talking to the developers, searching for any documentation, looking at the old system, learning about the new one, basically doing the work a BA would have done.

Once I had a solid understanding of the specifications I was working with, I started testing. I started with the simplest cases and the ones that I was most familiar with. You are probably thinking I should have done the opposite and started with the hardest, most complicated and risky transformations. If I was doing it over today, that is the approach I would take, but considering I was a complete rookie tester, I needed to start with a small, achievable task to build some confidence and establish my testing strategy.

My strategy was pretty simple. I gathered sample source data, ran it though the migration and verified the results on the other side. Then I started adjusting the source data, trying every scenario I could think of looking for ways to make the tools fail. As I worked through my list, reporting bug after bug, it became clearer to everyone just how large this task was. Expectations slowly became more aligned with reality and the measurement for success became more reasonable.

After a few months of work, we were ready to start migrating content. It wasn’t perfect and several bugs made it through, but they were minor and due to junk source data that we didn’t care about. The migration ran for months as different sites came online. The migration tools did their job and by all accounts, the testing aspect of the project was successful.

This experience gave me an unexpected appreciation for what testing is and how valuable it can be. It gave me confidence in myself that I could accomplish a task that I really had no business even attempting. It’s an experience that I can now look back on and be proud of.

Webinar – Should testers report every bug they find?

In December of 2012, Ryan Lamontagne and I got into a good discussion on the uTest forms about whether testers should report every bug they find. We decided to kick it up a notch and debate it live in a webinar!

http://forums.utest.com/viewtopic.php?f=13&t=4430

Webinar – Three more uTest Panel webinars

It’s been a busy past few weeks. In addition to picking up two new enterprise customer accounts (uTest TTL work) I was a panelist for three more uTest webinars.

Maximizing Your Benefit From The uTest Forums
http://forums.utest.com/viewtopic.php?f=55&t=4985

Maximizing Exploratory Testing Methods
http://forums.utest.com/viewtopic.php?f=55&t=4984

How to be a Quiet Tester That Customers Shout About
http://forums.utest.com/viewtopic.php?f=55&t=4986

 

Testing with the Microsoft Surface RT

About the Device

The Microsoft Surface RT is a interesting device to test with. It has the new Windows 8 interface, but it also still has the traditional desktop interface because not all things can be accomplished on the Windows 8 side. This can make things a little confusing, especially if you are just starting to test with this device. So, here are some instructions to help you get ready to set up and back to testing in no time.

Installing Apps

When we are testing apps that are in development, they are not available in the Windows Store so we will need to manually install them. Usually, you will be given a zip file that contains all the files you need.

  1. Download the zip file
  2. Go to your Desktop and locate the zip file (Should be in your Downloads folder)
  3. You can’t install directly from a zip file so you need to extract the files
  4. Long tap/release (or right-click if you have a mouse) to open the context menu
  5. Tap on ‘Extract All…”
  6. Select the location where you want the files saved to and tap ‘Extract’
  7. The extracted folder will open
  8. Locate the Windows Powershell Script (should have a Notepad icon)
  9. Long tap/release (or right-click if you have a mouse) to open the context menu
  10. Tap ‘Run with PowerShell’
  11. You’ll see a series of prompts, accept all of them
  12. If this is the first time you’ve installed an app, you’ll be required to sign up for a development license. It’s free and it has just a few simple steps
  13. After the application is installed, the PowerShell windows will just close. There is no indication that it was installed, but when you go back to your home screen, you should see the app on the far right

Uninstalling apps

  1. On the home screen, drag the app icon down just a bit until a check mark shows up just above the icon
  2. Release the app and the action menu will open from the bottom of the screen
  3. Tap ‘Uninstall’ to remove the app

Error logs

The Surface RT essentially is a PC, so application errors and crashes will be logged in the Event Viewer just as they are on a regular Windows desktop or lap top.

How you access the Event Viewer on the Surface RT is more complicated than on a regular PC since there is no Start button or search function. Here is how you can add a link to the Event Viewer to your desktop:

  1. From the Home screen, tap on ‘Desktop’
  2. Long tap/release (or right-click if you have a mouse) to open the context menu
  3. Select Personalize
  4. In the ‘Search Control Panel’, search for Event Viewer
  5. Tap on Administrative Tools(or Navigate to Control Panel > System And Security > Administrative Tools)
  6. Long tap/release (or right-click if you have a mouse) on Event Viewer to open the context menu
  7. Tap Send To > Desktop (create Shortcut)

Now that you have the event viewer open, you can view error messages and save the error files to attach to your bug reports.

  1. On the left pane, tap on Windows Logs >> Application
  2. On the right pane, tap on ‘Filter Current Log…’
  3. Select ‘Error’ and tap ‘OK’
  4. Select the error log(s) you want and on the right pane, tap ‘Save Selected Events’
  5. Save the events wherever you like and you can attach them to your bug reports

Taking Screenshots

Screenshots are pretty easy to take. You need to press the home button (Windows icon on the front of the Surface) and the down volume button (left side of the Surface) at the same time.
The file will be stored in your Photos > Pictures library > Screenshots folder

Taking Video

A well-created external video is usually the best way to go as I’ve done in all the above videos. Check out this post for some tips to improve your external videos.
I haven’t found a way to create mirrored videos yet, but I’ll update this if I do.

 

Webinar – Finding bugs in mobile devices

I was able to join Kayla Cox and Todd Smith for a uTest webinar to talk about testing mobile devices and how to find high-value bugs.

Since my microphone was terrible (and I might have been mumbling a little ) here is a summary of the points I made in our discussion.

Crashes
Understand that not all crashes are valuable.
Out of memory crash may be due to other apps using up 90% of your memory and the app you are testing just pushed you over the limit. The best way to know for sure, is to have a clean test bed. Restart your phone after you install a new app, and make sure no other apps are running in the background.

When you do get a memory related crash, use a memory management app to help you see where your memory usage spikes. Being able to identify a reproducible memory crash is usually a high-value bug

Connection Issues

  • Kill your connection while data is being transferred
  • Unplug your wi-fi router/modem
  • Turn on airplane mode
  • Turn of wi-fi on your device
  • Turn off cellular data on your device
  • Find places near you that have low or no signal and test there

Interaction with native and popular apps

  • Share something via email with no email set up
  • Log in using Facebook account with/without the Facebook app installed
  • Interrupt testing with phone calls, text messages, FaceTime calls etc
  • If the app changes the phone settings, make sure it does it correctly. Change it back manually in settings and see how the app responds

Investigation and Documentation
There are many topics on how to write good bug reports but there are a few points worth reiterating

  • Provide exact reproduction steps
  • Do root cause analysis – don’t report symptoms. I once saw 3 testers reported 3 different symptoms of the same bug. On the surface they all looked like different bugs, but a little analysis showed they were all caused by the same step they all overlooked. 

Bug Fix Failure Alerts Using TFS 2012 Scrum Process Template

My team is using Microsoft TFS 2012 Team System Web Access with the Scrum 2.2 Process Template as our Scrum management tool. We treat bug work items similar to how we deal with PBI’s, in that we assign tasks to the bug to complete the work. Usually this results in only two tasks: One development task to fix the bug, and one testing task to verify the fix and look for any related regressions around that fix.

When the developer has completed the bug-fix task, they use the task board to move their task from the ‘In Progress’ column, to the ‘Done’ column. But what should happen when the tester determines that the bug isn’t fixed? Well, the tester moves the development task back to the ‘To Do’ column and puts a note in the history field detailing the reason why the task wasn’t done.

The problem we were having is that the developers wanted to know when their bug fixes were not complete. Usually they would find out the next morning during our daily Scrum meeting, but they wanted to know right away. I solved this problem by using the Alerts feature available in TFS 2012.

  1. From your Team System Web Access site, click the settings button (the gear icon in the top right of the page)
  2. Select the ‘Alerts’ tab
  3. In the left navigation, select the ‘Team Alerts’ section
  4. Select ‘Work Item Alerts’
  5. In the main body of the page, click on ‘New…’
  6. In the ‘SELECT NEW ALERT TEMPLATE’ window, select ‘A change is made to a work item that is assigned to me’ and click ‘OK’
  7. Give the alert a more descriptive name
  8. Add 3 new clauses: Work Item Type = Task, State Changed From Done, State Changed To To Do
  9. Click OK

You’re all set. Your developers will start receiving emails when their bug tasks are moved back to the ‘TO DO’ column.

Here’s a quick video that walks you through the above steps:
How to add a bug-fix alert in TFS

Checking vs. Testing – The Problem with Requirement Documents

The prevalent idea that testers are dependent on a requirements document to do their job is a dangerous one. Requirements are not always needed to test. In fact, in many situations, they may actually reduce a tester’s effectiveness.

The process of deriving tests directly from the requirements has several names. The ISTQB uses the term “specification-based testing”, sometimes it’s referred to as “Happy Path” testing, but I think the most appropriate name is “checking”. Michael Bolton wrote a well-known post about this topic (http://www.developsense.com/blog/2009/08/testing-vs-checking/). Checking is confirming that what we believe is actually true. Products are built in accordance with the requirements, so the requirements are what we believe to be true. When we verify that our product meets the requirements, we are “checking” the product. When a tester relies on a requirement document to test, he isn’t testing, he’s checking.

When we test, we are exploring, investigating and learning. Our actions are influenced by new questions and ideas that haven’t yet been explored. The use of requirement documents while testing can cause problems because it can give a false sense of test completeness, it can steer testers in the wrong direction, and it can reduce the independent thinking of the tester.

Gives a false sense of test completeness

If we have verified that our product meets all of the requirements, does that mean that the product has been well tested? True, you have verified that the product behaves the way we expect it to (in specific situations) but you still don’t know how the product will behave in situations not specified in the requirements.

“That wasn’t in the requirements” I’ve heard testers use this excuse many times and it drives me crazy. As a tester it is your job to investigate the product to learn about areas and behaviors outside of the requirements. A product is much more than its conformance to the requirements. It’s up to you to cover that gap between what is expected of a product and what actually is.

Checking that a product meets the requirements is necessary of course, but checking alone does not indicate test completeness.

Steers testers in the wrong direction

When you first start testing a new product or feature, what should be the first thing you test? Some might say you should verify the requirements. I challenge that view. I think the requirements should be one of the last things you test. In my opinion, a developer who writes code that didn’t meet the requirements failed to do his job. Before the test team sees the product, the developer has already spent hours working and verifying that his work is correct. Although possible, a strong developer rarely produces work that doesn’t meet the requirements. With that in mind, there is little value in retesting his work, especially when you consider the other aspects of the product that haven’t been tested yet.

Requirement documents can stifle the creativity of exploratory testing. When testers have a requirements document in front of them, they may be more likely to verify the requirements first and focus on areas where the likelihood of learning new information is the lowest. Instead, they should focus their efforts on new tests and unexplored areas where the opportunity for learning is the highest. When testing without requirements, you eliminate its influence on your testing decisions; you have to rely on your own abilities, your knowledge, and your curiosity. You test.

All testers have heard the old saying “Trust but verify”. I’m not saying that we shouldn’t verify developer’s work, only that there’s more value in performing a test for the first time, then performing the same test multiple times. Focus on the activities that produce the most value first.

Reduces independence

The idea of “independence” (See ISTQB Foundation) refers to how close a person is to a product. A developer who wrote the code would have the least amount of independence, while a person in a different company who has never seen the product would have the most independence. Independence can often be a great quality for testers. Testers with no preconceived notions of how the product is supposed to work are able to view the product with more objectivity.

Consider the common situation where a product was built according to incorrect requirements. The tester was able to verify that the product met the requirements, so was there an issue? In this case the requirements served as a false crutch to the tester. Just because the product met the requirements doesn’t mean it was working correctly. A tester with no knowledge of the requirements would have been better positioned to identify any errors because he wouldn’t have been comparing it to an incorrect “truth”.

We have now seen some reasons why testers should avoid relying on requirement documents. While they may be a necessity for different “checking” activities, testers who wish to provide value thorough “testing” must understand that their value is best realized when they test without requirements.

Some Thoughts on Functional Testing

I wrote this as a Guest Blogger for the uTest blog.

There is an age-old expression that says “You only have one chance to make a first impression.” This is a hard truth in today’s world of instant gratification. If your product fails to deliver the first time, your customers will simply move on to the next thing. In-the-wild functional testing, as provided at uTest, is similar to a dress rehearsal for your application. Your application is exposed to a group of people who accurately represent your potential user base. They can identify and report the issues (that would have negatively impacted your customer’s first impression) before your customer ever has the chance to see them.

A functional tester has the ability to evaluate individual features of an application. They are familiar with typical application behavior and have the skills needed  to look objectively at a feature and see what’s wrong.

Perhaps even more valuable is a functional tester who is able to analyze individual pieces of an application within the context of the entire application. A functional tester looks at a particular item, identifies integration points between that item and other parts of the application, and then formulates a plan of how to inspect those touch points. Applications are usually weakest in places where different parts come together. A strong functional tester knows this and knows how to exploit those weaknesses to identify any lurking bugs.

Functional testing will only be successful if an organization’s underlying quality fundamentals are solid and everyone clearly understands how testing helps achieve the goals of the business. Functional testing is only one of many activities that collectively comprise a comprehensive testing strategy. Depending on the needs and expectations of your company, different testing activities such as performance, load, and security testing should be considered. Functional testing differs from other types of testing in that it most closely reflects the experience of the users. While performance affects the experience and security issues add risk to the experience – how the application functions IS the experience.