8 Tools I use to Accelerate My Testing

Inspired by Justin Rorhman’s post of a similar name with a slight twist focusing on tools that generally help accelerate my testing.

As a test and quality specialist embedded in an engineering team I have a lot of work to do on any given day. Our engineering team’s goal is to ship quality software on a regular basis to deliver value to our customers. Naturally I rely heavily on a number of tools to help me better understand what is going on with the software or code under test and/or to accelerate what I’m able to do in a given amount of time.

Like Justin said in his article “Testing tools don’t make me a good tester, but they do make me a better tester. These tools help amplify things I need to do repeatedly or give me access to the product I wouldn’t have otherwise. They also shape the way I think about testing problems, for better and worse.”

In no particular order these tools are:

  1. GitHub. We use the GitHub workflow to deliver all of our code including application, library, and test automation. With each push to a particular branch our CI system builds the application and runs our unit and integration tests. With each Pull Request I can see all or just specific changes, code reviews and test run summaries. All this helps keep our build pipeline clean, makes it easier to identify potential issues quickly and helps me quickly build my test strategy.
  2. Docker. Each separate application of our system has been dockerized (aka runs in a container). This makes it incredibly easy to set new people up (a few hours of setup instead of a day) and gives us some powerful logging for each application. It also reduces conflicts when switching between branches where libraries or decencies have been changed.
  3. Heroku review apps. As I mentioned above with each push our application is compiled through our CI system. Once finished our CI system pushes that complied build to Heroku which automatically provisions a staging like environment accessible through its own url. (This feature is called a Heroku Review application). In about 5 minutes after a set of changes have been built our team now has a brand-new test environment ready for those hard to test local problems like testing on real mobile devices or a few special integrated services. Being able to build these new automatic quickly allows our developers to show off / get quicker input to the work they are doing.
  4. Browser DeveloperTools. I primarily use Chrome and Safari but the developer tools in each browser are priceless. From Inspecting requests and looking at their data, to setting or removing experiment variables, debugging problems, looking for problems, profiling, responsive design, layout, checking DOM elements, etc browser DeveloperTools really are the first place to start. It would be so much harder to learn what is going on without them.
  5. CircleCI. Great for keeping your deployment pipeline clean, running your build tests and ensuring those changes get where they are going each time. It doesn’t much matter what CI system you use it’s helping to accelerate something that would normally take away from developer or tester time.
  6. EverNote. Surprising or not, I’m always taking notes about any number of things like quick terminal commands, operations for creating data, migrating databases in heroku, setting up configurations, test ideas, questions for follow up, etc. Anything I might forget goes here and I can get to it anywhere, any place on any device.
  7. iTerm2. A command line replacement utility with Oh My Zsh installed that I’ve customized and continue to customize with shortcuts and aliases to make my repetitive tasks quicker. Also customized with visuals showing which repo I’m in, what branch I’m on and if there are uncommitted files, because I do enough on the command line.
  8. Automated UI acceptance tests. Running our acceptance tests for each staging deploy (or set of deploys) allows us time to do more deep exploration on the application as a whole with a reduced focus on those areas covered by the existing tests.

Part of the desire for writing this was to record what my work looks like now and then be able to compare it later-on in the future. However, in the time since I started writing this article I changed jobs and now the future is here. Well, kind of. My new team’s development stack is a lot more varied, which means so are the tools I’m using. I think it’s still best to wait for half a year or so before I write a follow up post on the tools accelerate that work. In the meantime, what tools do you use that accelerate your testing?

Good and Bad UI Test Automation explained – Inspired by Richard Bradshaw’s Tweets

Generally speaking there’s a scary trend with the influx of people interested in test automation where (seemingly) everyone wants to automate at the UI level. For example the phrase “Test automation” seems to be synonymous with UI automation which seems to mean using Selenium. To be fair there are numerous reasons for this. First, if you primarily focus on functional testing through the UI, this application of automation is the most relevant and straightforward to understand. Second, there are a lot of resources out there (books, blogs, classes) for Selenium. Third it is much easier to write some blackbox automation at the UI level then to develop competence at writing automation at the API or unit level. (You should still learn to do this lower.)

So Richard took to Twitter to explain his thoughts. Let’s take a look:

Before we go very far, let’s look at what Richard means by targeted tests.

Tests are “targeted” if they are designed to reveal a specific risk or failure (we might call this risk-based). If you are doing risk-based testing, your tests should ALWAYS be designed this way, regardless of how you intend to execute the tests (automated or not).

Seams are how you use the system’s implementation to help you build more streamlined and realiable automated tests. Seams can be can be APIs, mocks, stubs, or faked dependecies of your real system.

Want some examples of Seams? Checkout this blog post by Angie Jones.

Yes, most times we can use better designed tests! Want to test the various ways to get to a particular feature? Use PATH testing. Want to test the feature itself? Use functional or domain testing, etc. Some test techniques will be better at surfacing one type of problem and bad at surfacing another. It’s not always necessary to design your UI tests as long feature rich Scenarios.

When we talk of interfaces and automating at the lowest level it is usually a good time to mention this is what the Test Automation Pyramid tries to describe. There are lots of versions of this but I recommend Alan Page’s Test Automation Snowman.

If you can find the failure by writing a test lower in the stack (at the API level intead of UI level or at the unit level instead at the API level), the faster it runs and the faster you get feedback. If you do have to write the test at a higher level like the UI can you find some way to decrease the setup time / data of the test? (See above for Angie Jone’s blog post.) If you aren’t sure maybe thinking about whether you are testing the UI or testing through the UI will help?

Yes! Adding testability like IDs, classes, data-test attributes, setting up test data, mocking, etc. should be done as quickly and dependbly as possible. If you can’t do this on your own (or you have a different process for making changes) bring it up during sprint planning in order to get the development teams help behind building a solution.

This IS a thing. Selenium is great, it’s open source and the WebDriver protocol is the soon-to-be de facto standard for interacting with browsers. However there are lots of tools, some will use Selenium under the hood and others that are completely different (think Cypress.io). Depending on the problem you need to solve, you may find a specific tool much better than Selenium. GitHub is a really good resource for finding open source tools and when you find something that might work, give it a try.

If you want to write Good UI Test Automation your tests should be ideally suited for the UI (and not some other layer), targeted, reliable, use seams where possible andcover a specific risk. If you aren’t here yet work to refactor your test code until you can get there.

Did I miss anything about Richard’s tweet storm that was important?

Opting out of A/B Tests while Running your Automated Tests

At Laurel & Wolf, we extensively introduce & test new features as part of A/B or split testing. Basically we can split traffic coming to a particular page into different groups and then show those groups different variations of a page or object to see if those changes lead to different behavior. Most commonly this is used to test if small (or large) changes can drive some statistically significant positive change in conversion of package or eCommerce sales. The challenge when running automated UI tests and A/B tests together is that your tests are no longer deterministic (having one dependable state).

Different sites conduct A/B tests differently depending on their systems. With our system we’re able to bucket users into groups once the land on the page which then either sets the AB Test and its variant options in local storage or as a cookie. Here are a few ways we successfully dealt with the non-determinism of A/B Tests.

Opting out of A/B Tests (by taking the control variant):

  1. Opt into the Control variant by appending query params to the route (e.g. www.kenst.com?nameOfExperiment=NameOfControlVariant). Most frameworks offer the ability to check for a query param to see if it should force a variant for an experiment and ours was no exception. For the tests we knew were impacted by A/B tests we could simply append to the base_url the query params for the experiment name and the control variant to avoid changing anything. This method didn’t require us to really care about where the experiment + variant were set (either as a cookie or localStorage) but really only worked for a single A/B test for a given test.
  2. Opt into the Control variant by setting localStorage. We often accomplished this by simply running some JS on the page that would set our localStorage (e.g.  @driver.execute_script("localStorage.setItem('NameOfExperiment', 'NameofControlVariant')")). Depending on where the A/B test sat within a test and/or if there were more than one, this was often the easiest way to opt out assuming the A/B test was set it localStorage.
  3. Opt into the Control variant by setting a cookie. Like the example for localStorage we often accomplished setting the cookie in the same way using JS (e.g.@driver.manage.add_cookie(name: 'NameOfExperiment', value: 'NameofControlVariant'). Again this was another simple way of allowing us to opt out of an experiment or even multiple experiments when necessary.

I know there are a few other ways to approach this. Alister Scott mentions a few ways here and Dave Haeffner mentions how to opt out of an Optimizely A/B test by setting a cookie here. How do you deal with your A/B tests? Do you opt-out and/or go for the control route as well?

Oh and if this article worked for you please consider sharing it or buying me coffee!

A typical day of Testing (circa 2018)

Recently I found myself repeatedly describing how I approach my testing role in a “typical day” and afterwards I thought it would be fun to capture some things I said to see how this might evolve over time:

Background

  • At Laurel & Wolf our engineering team works in 2 week sprints and we try to deploy to production on a daily basis.
  • We work within a GitHub workflow which essentially means our changes happen in feature branches, when those branches are ready they become Pull Requests (PRs), those PRs automatically run against our CI server, then get code reviewed and are ready for final testing by myself and our other QA.
  • All of our JIRA stories are attached to at least one PR depending on the repository / systems being updated.
  • Everyone on the engineering team has a fully loaded MacBook Pro with Docker containers running our development environment for ease of use.

 Daily

  • I’ll come in and if not already working on something, I’ll take the highest priority item in the QA column of our tracking system JIRA. (I try to evenly share the responsibility and heavy lifting with my fellow tester although I do love to jump into big risky changes.)
  • Review the GitHub PR to see /how big/ the code changes are and what files were impacted. You never know when this will come into play and at the very least it’s another source of input for what I might test. I will also do some basic code review looking for unexpected file changes or obvious challenges.
    • Checkout the branch locally and pull in the latest changes and get everything running.
  • If the story is large and/or complex, I’ll open up Xmind to visually model the system and changes mentioned in the story.
    • Import the acceptance criteria into as requirements. (An important input to testing but not the sole oracle).
    • Pull in relevant mockups or other sources cited either in the story, subtasks or other supporting documentation.
    • Use a few testing mnemonics like SFDIPOT and MIDTESTD from the Heuristic Test Strategy Model to come up with test ideas.
    • Pull in relevant catalogs of data, cheat sheets and previous mindmaps that might contain useful test ideas I might mix in. (When I do this I often update old test strategies with new ideas!)
    • Brainstorm relevant test techniques to apply based on the collected information and outline those tests or functional coverage areas at a high level. Depending on the technique (or what I brainstormed using mnemonics) I might select the data I’m going to use, make a table or matrix, or functional outline but it depends on the situation. All of these can be included or linked to the map.
  • If the story is small or less complex, I’ll use a notebook / moleskine for this same purpose but on a smaller scale. Frankly I also use the notebook in conjunction with the mind map as well.
    • I’ll list out any relevant questions I have that I want to answer during my exploration. Note any follow up questions during my testing.
    • I don’t typically time-box sessions but I almost always have a charter or two written down in my notebook or mindmap.
  • Start exploring the system and the changes outlined in the story (or outline in my mindmap) while generating more test ideas and marking down what things matched up with the acceptance criteria and my modeled system. Anything that doesn’t match I follow up on. Depending on the change I might:
    • Watch web requests in my browser and local logs to make sure requests are succeeding and the data or requests are what I expect it to be.
    • Inspect a page in DevTools for the browser or in React or Redux.
    • Reference the codebase, kick off workers to make something happen, manipulate or generate data, reference third party systems, etc.
    • Backend or API changes are usually tested in isolation and then with the features they are built to work with and then as a part of a larger system.
    • Look for testability improvements and hopefully address time during this time
    • Add new JIRA stories for potential automation coverage
    • I repeat this process until I’m able to find less bugs, less questions and/or I’m able to generate less and less test ideas that seem valuable. And/or I may stop testing after a certain amount of time depending on the desired ship date (unless there is something blocking) or a set time box.
  • When bugs are found and a few things can happen:
    • If they are small, I’ll probably try to fix them myself otherwise
    • I’ll let the developer know directly (slack message or PR comment) so they can begin working as I continuing to test
    • or I’ll pair with them so we can work through hard to reproduce issues.
    • If they are non-blocking issues I’ll file a bug report for addressing later
  • Run our e2e tests to try to catch known regression and make sure we didn’t break any of the tests.
    • This doesn’t happen often, our tests are fairly low maintenance.
  • Once changes are promoted to Staging I’ll do some smaller amounts of testing including testing on mobile devices. We do some of this automatically with our e2e tests.

Semi-Daily

  • Pickup a JIRA story for automation coverage and start writing new tests.
  • Investigate or triage a bug coming from our design community (an internal team that works directly with our interior designers).

circa 2018

I’m probably forgetting a ton of things but this is roughly what a “typical day” looks like for me. I expect this to evolve over time as data analysis becomes more important and as our automation suites grow.

What does your typical day look like?

Practice using Selenium Now!

Have you ever wanted to learn a little bit about Selenium WebDriver but didn’t know where to start?

Turns out there are some good tips / tutorials online for practicing writing Selenium in Ruby. One of those is a newsletter called Elemental Selenium that has something like 70 tips. You can sign up for the newsletter if you want but what I found valuable was to look at several of these tips, write them out (don’t copy + paste) and make sure you understand what they do. Turns out when you do this and you commit them to a repo, you can reference back to them when you come across similar problems in the future.

Simply stated [highlight]the goal is to:[/highlight]

  1. Read through the Elemental Selenium tips and then write (don’t copy + paste) the code yourself.
  2. Try running the tests locally and see how things work.
  3. Once you’ve written a few tests, refactor those example tests so they become more DRY (don’t repeat yourself). Create page objects.
  4. Commit these to your own repo on GitHub.

Putting your code on GitHub will have the benefit of showing you can write (at least) basic selenium automation. Although this code may not be your “best foot forward” given how new you’ll be, it is a starting point. As you learn more and make improvements, your code will reflect this.

These tips are (hopefully) grouped correctly but within each group there may be some variance. See if you can do one or two per day (some will be easier than others). If you see something interesting and want to jump to it immediately, go for it.

Beginning to Intermediate:

Intermediate to Advanced:

I’ve recommend a few people try this exercise because I found it valuable. Am I missing anything else? Has anyone else done something similar but in a different language or tool?

Installing GeckoDriver on macOS

Overview of naming conventions

  • GeckoDriver is the library you need to download to be able to use Selenium WebDriver with Firefox. These are the Selenium Bindings.
  • Marionette is the protocol which Firefox uses to communicate with GeckoDriver. Installed by default with Firefox.
  • FirefoxDriver is the former name of GeckoDriver.

Ways to install GeckoDriver:

  1. The easiest way to install GeckoDriver is to use a package manager like brew or npm such as npm install geckodriver. This method requires you some package manager installed but you probably should anyways.
  2. Run Firefox and GeckoDriver in a container using Docker. Simply download the combined container, start it and point your code at the right address. I’ve written about how to do this using Chrome, should be very similar to do Firefox.
  3. Specify it in your Selenium setup code. If you go this route, you can include ChromeDriver as well.
  4. Download the driver and add its location to your System PATH. These instructions are for Chrome but should work for GeckoDriver as well.

Once this is done, it should work like nothing has changed. The big advantage is you’ll now be able to use Firefox browsers newer than 46!

Selenium-WebDriver 2.53.x not working with Firefox 47 and beyond

The problem

I’m used to running selenium tests against Firefox locally (OS X Yosemite and now MacOS Sierra) both from the command line using RSpec and when using a REPL like IRB or Pry. I don’t use FF often so when I started having issues I couldn’t remember how long it had been since things were working. The problem was pretty obvious. The browser would launch, wouldn’t accept any commands or respond to Selenium and then error out with the message:

Selenium::WebDriver::Error::WebDriverError: unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055) from /usr/local/rvm/gems/ruby-2.1.2/gems/selenium-webdriver-2.53.0/lib/selenium/webdriver/firefox/launcher.rb:90:in `connect_until_stable’

This occurred for Selenium-WebDriver versions 2.53.0 and 2.53.4. It also seemed to occur for Firefox versions 47, 48, and 49.

The solution

Downgrade to Firefox 45.0 Extended Service Release (ESR).

I’m not the first one to post about the upcoming changes and lack of support for Firefox 47+. I probably deserve the error message for not paying more attention to the upcoming changes and will certainly look forward to implementing the new MarionetteDriver.

Installing SafariDriver on macOS

Safari + WebDriver aka SafariDriver comes included in Safari 10 which means as long as you have Safari 10 and later versions you can point your tests at Safari and run them without installing anything else.

Safari now provides native support for the WebDriver API. Starting with Safari 10 on OS X El Capitan and macOS Sierra, Safari comes bundled with a new driver implementation that’s maintained by the Web Developer Experience team at Apple. Safari’s driver is launchable via the /usr/bin/safaridriver executable, and most client libraries provided by Selenium will automatically launch the driver this way without further configuration. – WebKit Announcement 

That’s right, no brew installs and no system PATH setups!

Additional References:

Debugging Selenium code with IRB

Occasionally something will change in our system under test that breaks a Selenium test (or two). Most of the time we can walk through the failure, make some tweaks and run it again – repeating the process until it passes. Depending on how long it’s been since we last worked with the code, or how deeply buried the code is, it may not be enough to fix the test and we may have to tackle one or more of the underlying methods we used to build the code.

In these situations it can be helpful to debug our tests using an interactive prompt or REPL. In the case of Ruby we can use irb or Interactive Ruby to manually step through the Selenium actions, watching and learning. Here’s the general format for working with Selenium in irb; it’s very similar to how we code it:

  1. Launch interactive ruby: irb
  2. First, import the Selenium library: require 'selenium-webdriver'
  3. Second, create an instance of Selenium and launch the browser: driver = Selenium::WebDriver.for :firefox
    1. If you’ve got Chrome installed with an updated PATH, you can also swap firefox for chrome.
  4. Third, start entering your Selenium commands: driver.get 'http://www.google.com'

Here’s a simple example using irb:

In this example I’m bringing up Chrome as my browser and navigating to Google. I’m finding the search query box, typing my last name ‘Kenst’ and clicking enter. Thus an example of searching Google for my last name!

Technically once the browser is up we can navigate to whatever page we need without typing ALL of the individual commands. This is really valuable in those instances when you need to login, then navigate several pages before getting to the place you can debug. In other words do all the setup outside of irb, directly in the browser. Once you are in the proper location step through your code one command at a time (and lines with => show the responses to our commands, if any). Those responses will help us debug our tests by confirming what elements Selenium is picking up and what our methods are returning (if anything).

Additional References:

Review of The Selenium GuideBook: Ruby Edition

tl;dr If you’ve ever wanted to learn Selenium but didn’t know where to start, The Selenium GuideBook is the place (doesn’t matter which edition you use, it’ll be good).

 

Learning Selenium

The challenge of trying to learn Selenium WebDriver (simply referred to as Selenium) or any relatively new technology is that there is either too much disparate information (on the web), the material is out of date (many of the books I found) or of poor quality. With Selenium too many of the tutorials available to beginners had you using the Selenium IDE, which is a really poor option for writing maintainable and reusable test code. (See my previous post’s video for more on this.) I’ve walked out on conference workshops that sought to teach people to use the Selenium IDE to start their automation efforts. It wasn’t for me. I was going to do it right, I just had to figure out what that meant and where to start.

From the start I knew I wanted to learn about GUI test automation and more specifically Selenium WebDriver. I had tried WATIR (Web Application Testing in Ruby) and a few other tools but Selenium was open source and quickly becoming the go-to standard for automating human interaction with web browsers. It was and is the only choice.

Naturally I went searching the web for some tutorial or examples when I stumbled across several tutorials including Sauce Lab’s boot camp written by someone named Dave Haeffner. After struggling through the Bootcamp series (and finding some bugs in the process) I found Dave also produced a tip series called Elemental Selenium. I signed up for the Ruby weekly newsletter tips and went through many of the tips. Satisfied that Dave was worth learning from (good quality, relevant code examples) I decided it was time to try his book The Selenium GuideBook. I knew going into it, I was going to be the person maintaining the test suite and since I was more or less comfortable with Ruby I was happy The Selenium GuideBook came in that language!

Book Options

There are a few packages (book, code examples, videos, etc. ) for the language of your choice. As I said above I was more or less comfortable with Ruby so I ended up getting the “Ruby Edition, Just The Book” package. If I was doing this over today I probably would have done the “Cheat Sheets + Book” package and for JavaScript instead of Ruby.

The package itself contains a lot of great information and a number of materials:

  • The Selenium GuideBook; the Ruby edition is roughly 100 pages
  • Ruby code examples broken out by chapter
  • Elemental Selenium Tips eBook in Ruby
  • The Automated Visual Testing Guidebook

The first time I went through the book and code examples, it seemed redundant having different code for each chapter and example. It was only after I had gone through the chapters and examples for a second time, trying to apply and understand the differences that I began to understand the relative importance of seeing the code change chapter by chapter, example by example. The code examples all target an open source application Dave created called The Internet. It seems simple enough but many of the books and materials I went through either tried using Google or some badly written / hurried example.

The Book

Despite being less than 100 pages the Selenium GuideBook covers:

  • Defining a test strategy for your automation goals
  • Programming basics
    • Using page objects and base page objects to keep code clean
  • Locator strategies and tools
    • Relying on good locators seemed like a smart way to design tests. I wanted to avoid any record and playback tools and the poor locator strategies often employed.
  • How to write good acceptance tests
  • Writing Re-usable test code
  • Running on browsers locally and then in the cloud
  • Running tests in parallel
  • Taking your newly built tests, tagging them and getting them loaded into a CI server.

The whole package literally. In the preface Dave says the book is not full and comprehensive, it’s more of a distilled and actionable guide. The book really shows you how to start thinking about and building a test suite. It’s up to the user to take what they learned here and apply it to their application. That’s the fun part of the Elemental Selenium tips as well.

Applying the Book and Examples

After I had gone through all of the chapters and examples once, I went back through the relevant to me chapters and examples doing the following:

  1. Start with some very simple login tests.
    1. The book starts out this way as well. Writing tests that are not DRY or re-usable. but eventually get that way.
  2. Continuing through the code examples, getting a little more complicated and applying it to my own application.
    1. As I built out tests and start to see commonly repeated patterns, abstract out repeated code into relevant page objects. Eventually getting to a base page object.

In hindsight the hardest part of applying the book was trying to understand and apply a locator strategy within our application. While The Internet target application is great, it’s also a bit simplistic. Good for teaching, hard for bridging the sample application to the target application. Our single page .NET application was far more complicated and it took several attempts before I understood how my own strategy would work.

The transfer problem is always difficult. I mean how do you take what you learned and apply it to a slightly more sophisticated problem? It’s a difficult problem, not really a criticism of this book. It’s worth noting that whenever I had questions about what was written in the book, found a bug or two, or got stuck I could email Dave and within a week or so get a helpful response back.

Mission Accomplished!

With the lessons in this book and the Elemental Selenium Tips I was, through some focused time and lots of iterations, able to get a fairly good set of automated acceptance tests running against our application.

In other words, I highly recommend you buy the book. It’s slightly more expensive than similar books about Selenium but it’s far more effective. You are also directly supporting the author and with free updates and email tech support I think its well worth the cost.

Don’t believe me? Watch this video of Dave giving an overview of the GuideBook:

Additional References:

  1. http://www.tjmaher.com/2015/06/spotlight-on-dave-haeffner.html
  2. http://knowledge-anxiety.blogspot.com/2015/06/the-selenium-guidebook-and-thoughts-on.html
  3. http://davehaeffner.com/works/
  4. https://seleniumguidebook.com/