Do things that don’t scale

If you can find someone with a problem that needs solving and you can solve it manually, go ahead and do that for as long as you can, and then gradually automate the bottlenecks. – Paul Graham

With the title of Automation Engineer it might seem like automating bottlenecks is “my jam” and therefore “do[ing] things that don’t scale” might seem kind of an odd thing for me to agree with. However it turns out this is a useful way to think through solutions to problems, including testing problems.

From a business standpoint this makes a lot of sense, why waste time automating something until you are sure it has value? Many startups have blown through money building solutions to things (or automating problems) that aren’t actually important (or real problems). If you are an established company and thinking of branching into a new market segment, it’s ok to do everything by hand at first: signing up customers, placing orders, building website, etc.

From a software standpoint this also makes a lot of sense. Don’t worry about automating bottlenecks like test setup or execution until:

  1. You are sure the tests have value (business or technical) AND
  2. They’ve become bottlenecks to delivery

Otherwise don’t waste your time.

Appropriate Test Documentation & Formatting

The Question

Recently in an online forum a tester person asked:

Does someone have a simple example of test case (excel sheet) format? I am the only one tester in my company and we are trying to arrange the test documentation. Any advice or example will be useful.

I wish someone had given me this advice when I first started out:

Be careful about using someone else’s formatting and/or templates. If you don’t know what you are doing, you might be inheriting their issues and misunderstandings. It’s usually best to understand what you want to accomplish and then find a solution for yourself. Especially if it’s something simple like using Excel for documentation. Tables and matrices are great for organizing tests and don’t cost much money or time to make.

Appropriate Test Documentation

The appropriate documentation for your tests (or test cases) will depend to a large degree on the test technique and approach you use. Test Case Management systems (and similar systems) are often designed for functional or scenarios tests where the format is similar and you can list out multiple steps. Other test techniques such as Path Analysis, Domain Testing and even Risk Based testing can look very different when they are well designed. The same can be said when you decide to take an Exploratory approach to your testing where you are likely to write Exploratory Charters. Cramming all of these different techniques into one system usually isn’t the best idea.

The Smallest Amount

Focus on the smallest amount of documentation you need to do your job well. You have a finite amount of time and if you spend too much of it creating documentation you will have less time to do the work. If you are writing tests for other humans to read and understand, they should be written in a way that communicates this well such as using visuals (mind maps, diagrams, etc.) and summaries. Similarly if you are writing the tests for computers to read and understand they can (need to be) much more granular and step by step.

I’d wager most experienced engineers have gone through this process of: trying to document as much as possible but by the time you are done you didn’t do enough testing and the documentation isn’t done either. Maybe you put it all into a test case management system that no one else cares to read because it’s in this weird format that probably only makes sense to you.

Staying Agile

Focusing on the appropriate documentation for your tests and then creating the smallest amount you need to do your job well also allows you to have agility. Agile is all about being able to adjust to changes as they come in. Quick changes in direction aren’t kind on large amounts of pre-scripted documentation and/or can result in lost work.

I wish this approach was more obvious but a quick google search will reveal it’s not the dominate theory. However, if you read this and take the advice you’ll be learning from my mistakes. Focus on using the smallest amount of documentation you need to be successful and making sure that documentation is appropriate for your test choices will give you agility and future proofing even as you scale your team size and move into the future.

Selecting a few Platform Configuration Tests

I’ve been developing a GUI acceptance test suite to increase the speed of specific types of feedback about our software releases. In addition to my local environment I’ve been using Sauce Labs to extend our platform coverage (mostly by browsers and operating) and to speed up our tests by running more tests in parallel.

This pretty similar to what I consider traditional configuration testing – making sure your software can run in various configurations with variables such as RAM, OS, Video card, etc. Except on the web the variables are a little different and I am more concerned with browser differences than say operating system differences. Nevertheless, with over 700 browsers and OS platforms at Sauce Labs I still need to decide what configurations I start with and what configurations I add over time in the hope of finding failures.

Data

I figured the best place to start was with our current users and since the only “hard” data we had comes from Google Analytics, I pulled up the details of two variables (OS and Browser). Using TestingConference.org as a replacement for my company’s data, our most commonly used platform configurations include:

  • Browsers:
    • Chrome 47 & 48,
    • Firefox 43 & 44,
    • Safari 8 & 9, and
    • Internet Explorer 10 & 11
  • Operating Systems:
    • Windows 7,
    • Windows 8.1,
    • Windows 10,
    • Mac OSX 10.11 and
    • Mac OSX 10.10

Excluding a few constraints like IE only runs on Windows and Safari only runs on Mac, testing browsers and operating systems in combination could potentially yield up to 40 different configurations (8 browsers x 5 operating systems). Also, we designed our application to be responsive and at some point we probably want to test for a few browser resolutions. If we’ve got 40 initial configurations and 3 different browser resolutions that could potentially yield up to 64,000 different configurations. Realistically even for an automated suite and with one to two functional tests, that’s too many tests to run.

Reduce the number of tests

We can start with adding in those constraints I mentioned above and then focuses on just the variables we want to ensure we have coverage of. To get a better picture of the number of possible configuration tests I used ACTS (Automated Combinatorial Testing for Software) tool and based my constraints on what Sauce Labs has available today for configurations. After I added OS and Browsers it looked something like this:

ACTS Window

If I wanted every browser and operating system in combination to be covered (all-pairs) then according to ACTS there aren’t 40 different configurations, just 24 configuration options. That’s more manageable but still too many to start with. If I focus on my original concerns of covering just the browsers, I get back to a more manageable set of configuration options:

Test OS Browser
1 Windows8.1 Chrome47
2 Windows10 Chrome48
3 OSX10.11 Firefox43
4 Windows7 Firefox44
5 OSX10.10 Safari8
6 OSX10.11 Safari9
7 Windows10 IE10
8 Windows7 IE11

8 configuration options are way more manageable and less time consuming than 24 options, considering each one of those configurations will each run a few dozen functional tests as well.

A Good Start

Selecting the configurations is probably the easiest part of configuration testing (and the only thing I’ve shown) but I’ve found its worth thinking through. (The harder part is designing the automated acceptance tests to produce useful failures.) Using ACTS at this point may seem like overkill when we could have just selected the browsers from the beginning but it didn’t take much time and should make it easier in the future when we want to add more variables or change the values of our existing variables.

Exploratory Charters in GitHub

GitHub Issues
GitHub Issues (for TestingConferences.org)

Since CAST 2015 I’ve wanted to implement an interesting idea that could potentially give my testing greater visibility and greater scrutiny: Putting exploratory testing charters into our project tracking tool.

At work we use GitHub to host our code which means we use GitHub Issues as our bug tracker. Then we use ZenHub on top of GitHub Issues as project management / tracking tool. As the sole (omega) tester on the development I use GitHub issues for a number of activities:

  • Filing and reviewing issues (bugs and enhancements)
  • Occasionally reviewing pushed code changes referenced in the issues I’m tagged on
  • Occasionally reviewing pull requests
  • Committing the rare change to our production code (I’ve done this once so far)
  • Committing code changes to our test automation repo

Do charters belong?

Putting my exploratory charters into GitHub Issues (so they’d be visible in both GitHub and ZenHub) seemed like a worthy proposition. On the one hand it would allow my charters and subsequent notes to be seen and commented on by the whole team. On the other hand, would they really belong? GitHub Issues can be used to track tasks, bugs and enhancements, it’s a pretty simple and straightforward difference. My teammates have been creating tasks for a while and in the last few months I’ve been adding charters.

So far, no one but me seems to care. Great! First step complete. I already break the charters up into test sessions with notes and debrief information. After each session that information usually stays in my session notes. Instead it should flow into it’s corresponding charter’s task so I have a stream of updates as to what this charter has uncovered and what there’s still left to do.

That’s my next step, use the GitHub task to more actively record the details of the session and report back how the charter is going. From there I can keep track of the amount of effort I’ve spent (ZenHub allows us to track hours assigned to a task) per charter. Once this becomes more comfortable, the final step is to get our project management team to help influence the prioritization and maybe creation of those charters. Small steps! Shout out to Jesse Alford for the suggestion!

And nothing else funny happened

I was recently talking with someone about their testing strategy and process when I noticed they were trying to build overly-detailed test scripts (procedures). It didn’t take them long to realize specifying to such detail often left them bored (writing became redundant) and so each test became less and less detailed. I offered to take a look at their tests to see if I could help improve things and as I saw, what I consider to be, their “typical” scripted tests with each line having a step and expected result I started thinking of something Pete Walen once said:

…and nothing else funny happened.

Peter Walen, STPCon

Pete Walen dropped this gem at STPCon in San Diego a few years ago during Matt Heusser’s talk Where do Bugs Come From? Matt had just shown the Moonwalking Bear video:

I’d guesstimate at least half of the packed room hadn’t seen this. The discussion turned to scripted testing and the inherent problem of in-attentional blindness. Pete shared a past experience where he told some testers he was working with to include the phrase in the expected results of their scripted tests. It’s a kind-of-heuristic to help someone remember, even though they are focusing on one thing at depth, they need to be aware of the potentially many other things going on so that “nothing else funny happens”.

It was such a simple, powerful idea it continues to stick with me, especially when I see someone trying to specify “most everything” in their scripted tests.

As it happens, this post coincides with STPCon 2015 back in San Diego right now. There looks like a lot of awesome speakers (Matt Heusser, Michael Larsen, and Smita Mishra) but I’m particularly intrigued by Andy Tinkham’s High Volume Automation in Practice and Dave Haeffner’s Using Selenium Successfully. I wonder, is anyone live blogging?

How do you handle regression testing?

Matt Heusser sent the context-driven-testing email group a series of questions about handling regression testing. Specifically he asked:

How do your teams handle regression testing? That is, testing for features /after/ the ‘new feature’ testing is done. Do you exploratory test the system?  Do you have a standard way to do it? Super-high level mission set?  Session based test management? Do you automate everything and get a green light? Do you have a set of missions, directives, ‘test cases’, etc that you pass off to a lower-skill/cost group? (or your own group)? Do you run all of those or some risk-adjusted fraction of them?  Do they get old over time? Or something else? I’m curious what your teams are actually doing.  Feel free to reply with /context/ – not just what but why – and how it’s working for you. 🙂

Matthew Heusser

My Response:

I worked for a small software company where I was the only tester among (usually) 3-4 developers. Our waterfall-ish development cycles were a month+ in development with the same time given to testing. After the new features were tested, if we had time to do regression testing it was done through exploratory testing at a sort of high level. I don’t think I ever wrote out a “standard way” to do it but it fit into my normal process of trying to anticipate where changes (new features, bug fixes) might have affected the product. If I had understood SBTM at the time I would have used it.

We’ve never gotten around to automating regression testing. A part of that has to do with time constraints – small company I wear multiple hats, could have managed my time better, etc. Other reasons involve not really knowing how to approach designing a regression suite. I’ve used Selenium IDE in the past but automating parts of our web applications GUI isn’t possible without changes.

When I’ve had test contractors we used a set of missions to guide the group so everyone could hit parts of the application using their own understanding and skill (although this was our approach I don’t think it was necessarily an understood approach =).) In all of the testing / regression or otherwise its all based on some sort of risk – based fashion.

I have a lot to learn

Mostly I feel like I don’t know enough about automated testing and how to change my testing approach to include an appropriate amount of automation. It seems reasonable to assume some automated regression tests could help provide some assurance in the build not changing for the worse (at least for the areas you’ve checked). Although I continue to commit time to learn more about testing I haven’t committed much time to learning about automation and I believe its to my detriment. I guess I know where to focus more.

Throw someone else in to help QA it faster!

“Throw someone else in to help QA it faster!”

A former boss (or two) of mine

I’ve heard this statement many times in my career but it happened again just recently and it got me thinking. Aside from the poor choice of words, about “QAing” something (is that really a verb?), why would someone say this?

This seems to happen when management realizes it will take longer to test something than they initially planned and/or some client demands a product sooner. The most recent occurrence came when management didn’t communicate the expected release date and freaked at the estimated test duration. My response was you can have the product whenever you want but let me tell you what won’t be tested. This elicited the response “no we don’t want to not test it, how about we… throw someone else in to help QA it faster.” Clearly someone hasn’t heard of Brook’s law.

Adding manpower to a late software project makes it later.

Fred Brooks

Brook’s Law is a term coined by Fred Brooks in his book The Mythical Man-Month which states “adding manpower to a late software project makes it later”. It also appears the person saying this doesn’t understand what Quality Assurance (or QA) means.

If the role of software testing is to help the team understand vital information about the product and you bring in someone who doesn’t understand how this is accomplished, the value both will be providing is diminished. You slow down the primary tester as they coordinate with the person being brought in as work is divided up based on skill and comfort level. Configurations of different machines, problems local to the users and a whole host of other problems can crop up as well. In other words it takes time for people to become productive on a project.

Anyone who does time-based work (has a deliverable) can tell you it’s helpful to know when you need to be done. Crazy, right? Testing a product is a never ending journey but specific periods of testing can end, for example when the product is needed in the field. There will always be more testing to do but you don’t always have time nor is it always a good use of resources. Dates can help. If this statement comes up often either Management or the Team has problems communicating with each other about when things need to be done. Setting dates isn’t a sure fire method since dates can change but so can the decision on what needs to still be tested and what’s acceptable risk for the field.

While it’s possible to get an additional person to add some incremental value into a project (they might have some unique perspectives that can help, especially if they are subject matter experts) it’s going to take them awhile. Don’t assume “throwing someone else in” will do anything other than make the testing duration longer.

Rapid Testing Intensive Confirmed!

(Stolen from the Rapid Testing Intensive site)

It’s official I’m booked for the onsite Rapid Testing Intensive with James and Jon Bach at the end of July on Orcas Island in Washington. According to the website this testing intensive will be based on “… Session-Based Test Management and Rapid Software Testing methodologies” and will “…allow you to see how the modern theory of testing meets practical work.” Sounds like a blast.

There are 10 onsite and 42 online participants as of 4/2/12 and one of those onsite partcipants is Robert Sabourin. I was in his “Using Visual Models for Test Case Design” class last year at StarWest so it will be interesting to work side by site with him as well as a few of the other participants.

As I said in my prior post my goal is for: “Experience and feedback on modern testing methodologies!” Can’t wait.

5 Ways to Revolutionize your QA

I can’t remember where I originally found this post and the corresponding eBook but the eBook is definitely worth taking a look at. Here is the former uTest blog post, now Applause blog post.

The 5 ways or insights are:

  1. There are two types of code and they require different types of testing
  2. Take your testing down a level from features to capabilities
  3. Take your testing up a level from test cases to techniques
  4. Improving development is your top priority
  5. Testing without innovation is a great way to lose talent

In point 2, James Whittaker also talks about a planning and analysis tool he used at Microsoft called a CFC or Component – Feature – Capability analysis. This allowed them to take testing down from features to capabilities.

The purpose is to understand the testable capabilities of a feature and to identify important interfaces where features interact with each other and external components. Once these are understood, then testing becomes the task of selecting environment and input variations that cover the primary cases.

While this tool was designed for testing desktop software I’m inclined to think it would work well for testing web applications. Essentially with the CFC you are mapping out the individual components / features in the web application in a branching form that closely resembles a mind map. Matter of fact a mind map might be better! =)