Selecting a few Platform Configuration Tests

I’ve been developing a GUI acceptance test suite to increase the speed of specific types of feedback about our software releases. In addition to my local environment I’ve been using Sauce Labs to extend our platform coverage (mostly by browsers and operating) and to speed up our tests by running more tests in parallel.

This pretty similar to what I consider traditional configuration testing – making sure your software can run in various configurations with variables such as RAM, OS, Video card, etc. Except on the web the variables are a little different and I am more concerned with browser differences than say operating system differences. Nevertheless, with over 700 browsers and OS platforms at Sauce Labs I still need to decide what configurations I start with and what configurations I add over time in the hope of finding failures.


I figured the best place to start was with our current users and since the only “hard” data we had comes from Google Analytics, I pulled up the details of two variables (OS and Browser). Using as a replacement for my company’s data, our most commonly used platform configurations include:

  • Browsers:
    • Chrome 47 & 48,
    • Firefox 43 & 44,
    • Safari 8 & 9, and
    • Internet Explorer 10 & 11
  • Operating Systems:
    • Windows 7,
    • Windows 8.1,
    • Windows 10,
    • Mac OSX 10.11 and
    • Mac OSX 10.10

Excluding a few constraints like IE only runs on Windows and Safari only runs on Mac, testing browsers and operating systems in combination could potentially yield up to 40 different configurations (8 browsers x 5 operating systems). Also, we designed our application to be responsive and at some point we probably want to test for a few browser resolutions. If we’ve got 40 initial configurations and 3 different browser resolutions that could potentially yield up to 64,000 different configurations. Realistically even for an automated suite and with one to two functional tests, that’s too many tests to run.

Reduce the number of tests

We can start with adding in those constraints I mentioned above and then focuses on just the variables we want to ensure we have coverage of. To get a better picture of the number of possible configuration tests I used ACTS (Automated Combinatorial Testing for Software) tool and based my constraints on what Sauce Labs has available today for configurations. After I added OS and Browsers it looked something like this:

ACTS Window

If I wanted every browser and operating system in combination to be covered (all-pairs) then according to ACTS there aren’t 40 different configurations, just 24 configuration options. That’s more manageable but still too many to start with. If I focus on my original concerns of covering just the browsers, I get back to a more manageable set of configuration options:

Test OS Browser
1 Windows8.1 Chrome47
2 Windows10 Chrome48
3 OSX10.11 Firefox43
4 Windows7 Firefox44
5 OSX10.10 Safari8
6 OSX10.11 Safari9
7 Windows10 IE10
8 Windows7 IE11

8 configuration options are way more manageable and less time consuming than 24 options, considering each one of those configurations will each run a few dozen functional tests as well.

A Good Start

Selecting the configurations is probably the easiest part of configuration testing (and the only thing I’ve shown) but I’ve found its worth thinking through. (The harder part is designing the automated acceptance tests to produce useful failures.) Using ACTS at this point may seem like overkill when we could have just selected the browsers from the beginning but it didn’t take much time and should make it easier in the future when we want to add more variables or change the values of our existing variables.

When to use a Gemfile

I’ve been building a GUI acceptance test automation suite locally in Ruby using the RSpec framework.  When it was time to get the tests running remotely on Sauce Labs, I ran into the following error:

RSpec::Core::ExampleGroup::WrongScopeError: `example` is not available from within an example (e.g. an `it` block) or from constructs that run in the scope of an example (e.g. `before`, `let`, etc). It is only available on an example group (e.g. a `describe` or `context` block).
occurred at /usr/local/rvm/gems/ruby-2.1.2/gems/rspec-core-3.2.2/lib/rspec/core/example_group.rb:642:in `method_missing'

It took a few minutes debugging before I spotted the error:


Source of problem: My remote tests were using a different version of RSpec than I was locally. Solution: Create a Gemfile to specify the version of using Rspec I’m using.

Since I didn’t realize I needed a Gemfile my question was, in general, when should someone use a Gemfile? According to the manual, a Gemfile

describes the gem dependencies required to execute associated Ruby code. Place the Gemfile in the root of the directory containing the associated code.

For example, in this context, I would place a Gemfile into any folder where I specifically want to call tests to run. In my case that meant a few specific locations:

  • At the root of the folder – where I run the whole suite of tests
  • In the /spec/ folder – where I typically run tests at an individual level

At a minimum I’d specify:

  • A Global Source
  • Each Gem I use locally that Sauce Labs will need to use

In the end it might look something like this:

Test, adapt, and re-test.

Installing Ruby and Watir on Windows 7

I recently started playing around with Web Application Testing in Ruby or Watir for short. As is recommended, I downloaded the Homebrewer’s Guide to Watir by Zeljko Filipin and started trying to install the latest versions of the RubyInstaller for Windows (currently 2.0.0) when I ran into a few problems. (I like the idea of using Leanpub to create and modify install documents for open source applications.)

I installed Ruby and Watir on a Windows 7 64 bit machine using the 32 bit versions and everything seems to work fine so far. Here’s how I installed everything using the RubyInstaller and instructions from the above guide (note: some of these instructions will be a duplicate of the guide):

Ruby Installation

  1. Open a command prompt (Run and type cmd) and type ruby -v to see if you have Ruby installed (if you don’t it will say the command isn’t recognized) and the version number.
  2. I have Windows 7 64 bit but I used the 32 bit installers from RubyInstaller Downloads including:
    1. Ruby 2.0.0-p247
    2. DevKit-mingw64-32-4.7.2
  3. Run the Ruby installer and on the optional tasks page select the below items before completing:
    1. Add Ruby executables to your PATH
    2. Associate .rb and .rbw files with this Ruby installation
  4. Open a new command prompt and type ruby -v to see if Ruby installed and the version number.
  5. (Optional) Once installed you can update the RubyGems distributed with the installer by typing gem update --system and watching for the “RubyGems system software updated” to know when its complete.
  6. Move on to the DevKit installation.

DevKit Installation

  1. Run the DevKit installer but change the extraction folder to C:devkit
  2. Open a command prompt (Run and type cmd) and change the folder to C:devkit (use the command cd c:devkit).
  3. Run the command ruby dk.rb init. If this step is successful you’ll see a response like “[INFO] found RubyInstaller v2.0.0 at C:/Ruby200”.
  4. Run the command ruby dk.rb install. If this step is successful you’ll see a response like [INFO] Installing C:/Ruby200/lib/ruby/site_ruby/2.0.0/
    and [INFO] Installing C:/Ruby200/lib/ruby/site_ruby/devkit.rb
  5. Move on to Watir installation.

Watir Installation

This is where I ran into problems with the 2.0.0 version of Watir. Something about the mini_magick version erroring out. To prevent this problem we do:

  1. Run the command (still from the C:devkit command window) gem install mini_magick -v 3.5.0 which works around the version problem. You should get a response like “2 gems installed”
  2. Then run the command gem install watir --no-ri --no-rdoc to install the rest of Watir. You should get a response like “Successfully installed watir-4.0.2-x86-mingw32 and 23 gems installed”.

Check the installation

In the same command window type (after each line of code submit the commands)
  1. irb
  2. require "watir"
  3. browser =
  4. browser.goto ""

If all is setup correctly you should get your default browser to open a new window and then browse to Google. Good luck.

Note: These steps weren’t meant to replace the Homebrewer’s Guide to Watir by Zeljko Filipin instructions but to improve them for the 2.0.0 release. For everything else please refer to the guide.

First principle reasoning

When I was young I remember wanting to be an awesome football player like Joe Montana, or an FBI agent working on the X-Files like Fox Mulder. These days I want to have the skills to identify and solve problems like Elon Musk.Musk is an interesting person. He’s created and sold numerous companies and with the profits he’s created a rocket building / space exploration company that is now the first (private) company to make it to space. He’s also built an American electric car company. While all these things make Musk an interesting person on the surface, its his approach that makes him enviable.In his TED talk Musk credits his training in physics with his ability to see and understand difficult problems. He says physics is about how to discover new things that might seem counterintuitive and physics first principle provides a good framework for thinking. The video is a good conversation between Elon Musk and TED curator Chris Anderson, I recommend watching it. Musk mentions first principle reasoning at about 19:37:

According to Wikipedia first principle in physics means you start directly at the lowest levels – at the laws. Musk provides a slightly easier understanding saying first principle is reasoning from the ground up as opposed to reasoning by analogy, which is copying what other people do with slight variations.

Musk further elaborates on first principle reasoning in this video. To summarize he says you look at the fundamentals of things, make sense of it, construct your reasoning and conclusion and (if possible) compare that to whatever is the current understanding. A part of that process involves questioning conclusions, asking whether or not something could be true. Sounds like Musk’s constantly modeling, learning, testing and re-modeling.

In thinking about my job there seems to be more reasoning by analogy than perhaps there should be (or at least its obvious to someone new). Whenever one of my developers or I ask why something is, why some conclusion has been reached – the typical response is “that’s how its done”. If I ask my test team why they do something a certain way its always “that’s how we’ve always done it” and there seems to be no desire (at least I haven’t seen it yet) to know whether something makes sense or is based on a real understanding of the problem. Perhaps there should be more modeling, learning and testing?

We all do some reasoning by analogy, in many ways it’s a much simpler way to communicate and learn but for many of us in the software engineering fields (testers and developers) perhaps we confuse reasoning methods? So how do we determine when we need to use first principles and when its ok to use analogy in reasoning? That’s the million-dollar question. I think we do like Musk: create a model, ask questions to help us learn, test and when we aren’t satisfied with the answer, we reason from the ground up.

uTest’s Business Model

Note: I am an active uTester.

The last few months I’ve completed a number of rounds of testing for uTest’s clients, mostly in dealing with web applications for my iPhone. In fact a majority of work I’ve done since joining has been for functional testing of mobile applications. It’s been fun because mobile testing isn’t in my area of expertise but is a nice break from my normal routine and I like learning new things.

uTest’s Business Model:

A few months ago I was talking with my boss about new options for helping me test our software. I work for a small company where I’m the only tester. Often the backlog for getting our releases out is me. My boss was talking about adding an offshore resource and I brought up the idea of uTest and their crowdsource model. He thought it was an interesting idea and so he contacted uTest to get more information.

A few weeks later we had a quote from uTest and a chat with one of the sales reps which gave me a interesting perspective into their business model. uTest prefers to sell their services in packages which generally include several rounds of testing (the time between rounds is up to you). The sales engineer’s try to get an understanding of your testing needs and then give you a flat price per round with a minimum number of rounds plus a monthly Software as a Service (SaaS) charge for access to uTest’s application – a must have for the tester’s to submit bugs, test cases, etc. I think our application was considered pretty big / complex so for 3 rounds it was just north of $7,000.

That means we’d pay uTest $7k upfront plus the SaaS access charge each month. From there we’d work with a project manager and tell them how many tester’s we need, what type of backgrounds tools they need, etc. Then the project manager builds the test process and plan with you. Essentially you are hoping you get a good project manager otherwise the money you drop and the test outcomes may not be worth it. The actual payments the tester’s receive (for test plans and bugs) comes from uTest out of that flat fee.

Makes me wonder what the average payout to tester’s are per round of testing? Probably less than $1k depending on the size. That means a majority of the money is going to pay for your project manager and to uTest’s wallet.