Good and Bad UI Test Automation explained – Inspired by Richard Bradshaw’s Tweets

I like the idea of blog posts as reactions to other material, be it articles, blogs or tweets. Richard did this for Alan Page’s tweets. Now I’m doing it for his.

Richard Bradshaw went on a Twitter deep dive last month about UI Test Automation and some of the subtlties involved in doing it well and not so well. I think it’s an excellent topic to highlight.

Generally speaking there’s a scary trend with the influx of people interested in test automation where (seemingly) everyone wants to automate at the UI level. For example the phrase “Test automation” seems to be synonymous with UI automation which seems to mean using Selenium. To be fair there are numerous reasons for this. First, if you primarily focus on functional testing through the UI, this application of automation is the most relevant and straightforward to understand. Second, there are a lot of resources out there (books, blogs, classes) for Selenium. Third it is much easier to write some blackbox automation at the UI level then to develop competence at writing automation at the API or unit level. (You should still learn to do this lower.)

So Richard took to Twitter to explain his thoughts. Let’s take a look:

Before we go very far, let’s look at what Richard means by targeted tests.

Tests are “targeted” if they are designed to reveal a specific risk or failure (we might call this risk-based). If you are doing risk-based testing, your tests should ALWAYS be designed this way, regardless of how you intend to execute the tests (automated or not).

Seams are how you use the system’s implementation to help you build more streamlined and realiable automated tests. Seams can be can be APIs, mocks, stubs, or faked dependecies of your real system.

Want some examples of Seams? Checkout this blog post by Angie Jones.

Yes, most times we can use better designed tests! Want to test the various ways to get to a particular feature? Use PATH testing. Want to test the feature itself? Use functional or domain testing, etc. Some test techniques will be better at surfacing one type of problem and bad at surfacing another. It’s not always necessary to design your UI tests as long feature rich Scenarios.

When we talk of interfaces and automating at the lowest level it is usually a good time to mention this is what the Test Automation Pyramid tries to describe. There are lots of versions of this but I recommend Alan Page’s Test Automation Snowman.

If you can find the failure by writing a test lower in the stack (at the API level intead of UI level or at the unit level instead at the API level), the faster it runs and the faster you get feedback. If you do have to write the test at a higher level like the UI can you find some way to decrease the setup time / data of the test? (See above for Angie Jone’s blog post.) If you aren’t sure maybe thinking about whether you are testing the UI or testing through the UI will help?

Yes! Adding testability like IDs, classes, data-test attributes, setting up test data, mocking, etc. should be done as quickly and dependbly as possible. If you can’t do this on your own (or you have a different process for making changes) bring it up during sprint planning in order to get the development teams help behind building a solution.

This IS a thing. Selenium is great, it’s open source and the WebDriver protocol is the soon-to-be de facto standard for interacting with browsers. However there are lots of tools, some will use Selenium under the hood and others that are completely different (think Depending on the problem you need to solve, you may find a specific tool much better than Selenium. GitHub is a really good resource for finding open source tools and when you find something that might work, give it a try.

If you want to write Good UI Test Automation your tests should be ideally suited for the UI (and not some other layer), targeted, reliable, use seams where possible andcover a specific risk. If you aren’t here yet work to refactor your test code until you can get there.

Did I miss anything about Richard’s tweet storm that was important?