Upgrading to WebDriverIO 6

In March I went through the process of upgrading to WebDriver v5. Last month I took the next step by upgrading our deployment to version 6 so we’d be current. I learned quite a bit from that first upgrade which made this upgrade a whole lot easier.

Here’s what I did to upgrade to WebDriverIO 6:

  1. Checked the Docs. I scanned through the change log sections on breaking changes. Maybe I’m boring but nothing in the docs breaking changes notes looked to impact our setup! 👏
  2. npm oudated . This is one of my favorite (and mostly unused) npm commands. It told me specifically what packages I had that were outdated. Hint: it was all of them. 🤯
  3. Upgrade the easy stuff.
    1. Based on what npm returned, I began by updating my package.json file for 3rd party libraries. Libraries such as prettier, chromedriver, moment, etc. that all played a part in my solution but didn’t deal directly with my tests. Then I npm install the latest changes.
  4. Run the whole test suite looking for failures.
  5. Upgrade the harder stuff.
    1. Same thing as step 3, updated the package.json versions to be the “latest” and then npm install.
    2. Another way to do this is to remove the node_modules folder and re-install each package based on the install instructions.
  6. Run the whole test suite… until nothing is broken!

More reasonable!

This upgrade wasn’t as daunting as I initially feared. With upgrades there’s always some level of concern you are going to “ruin a good thing”. Once I moved past that concern, there wasn’t much effort involved in getting things to work. Lots of credit goes to the WebDriverIO team for making the process more reasonable and straightforward, which of course they stated in their announcement:

This major update is much more reasonable and contains subtle changes that will help the project further grow while remaining performant at the same time.

I’ve been pretty happy thus far using WebDriverIO and I’m excited to see where things go!

FAQs to How I Became An Automation Engineer (talk)

I’ve given the talk on How I Became an Automation Engineer a few times. Each time I’ve gotten good feedback and a lot of questions from engaged participants. With each repeated question, I try to update my talk to address those points for future audiences. However often I do this some repeated questions still arise. I’ve taken this challenge as both an excuse to write out my answers and try out a FAQ schema block!

Frequently Asked Questions

Here are the FAQs for my talk How I Became an Automation Engineer.

How do you decide the scripting or programming language to use in your testing? Do you typically use the same language the app is built in?

My recommendation is to use a programming language that your company supports. By this I mean a language that someone within your company uses and can provide feedback on your code through code reviews. This may or may not be the language the application is built in. Most companies use several languages. You might build your tools in JavaScript while the application is built in Java.

Ultimately your goal is to tackle the task at hand. Use the tools and languages that make sense.

Is (insert name of your current tool) sufficient enough to perform all types of automation testing?

No, probably not. Hopefully you work for an organization that wants to use the right tool for the right job.

Part of the process for learning about tools is we figure out what they are and aren’t good for. No one tool is good for all jobs. The same goes for the person using the tool. If you only know one tool you’ll try to use it for every problem. The more tools we learn, the more likely we will use the right tool for the right job. This also makes us more valuable in the market for new jobs.

What are your thoughts on using frameworks with record features to start learning automation?

It depends on your needs and your skills.

In the past, Record and Playback features were sold as Snake Oil. They’d fix all your problems. But they usually were much good beyond proof of concepts.

Today, more and more services are cropping up with codeless / record and playback capabilities. If these meet the needs of your project and/or company, then use them. One challenge for the individuals learning and using these these tools is they might not benefit your (market) value in the long term.

Sometimes the team or project doesn’t support automation either for budget or time concerns. How do you tackle that?

Test Automation is there to provide feedback, but it costs money and takes time to build and maintain. So if you are on a short term project then it might not be worth the time an effort to do certain kinds of test automation.

What kinds make sense? Well it depends on the project. It might be worth it to do unit tests but not much else. Or it might be worth it to build some one-off probabilistic automated tests to help you discover problems because it doesn’t have to be maintained. Remember the point of automation is to reduce our time or extend our capabilities but all that must be done within the context of your project.

Do you recommend learning to test and automate at the API level? Is Postman adequate for this?

Yes! In general I recommend automating tests at the unit level first, then at the service or API level and lastly at the UI level. Learning to explore APIs is a big part of this and eventually once you explore them, you can create automated tests.

Postman is a good tool for exploring APIs (there are probably others). It might also make sense for you to write some automated tests in Postman but that really depends on your goals and/or if there are better places to do it. This depends on your overall strategy.

According to your last slide, the future of the Automation Engineer role is uncertain. Do you think it’s better to move to the developer role or there is still hope that automation roles will be still exist?

We know that in high performing teams developers own the automated tests and testers are allowed to explore for risks. So there’s a high probability in companies looking to build high performing teams that Automation Engineering roles will disappear or not exist. There’s also a high probability of developer roles being needed in the future. We just don’t know when any of this will occur. If you are going to move, it makes sense to survey the landscape and see what the options are, see what your skill levels are and be practical.

What is the name of the book about you recommend / quote about high performing teams handling test automation?

Want to watch the latest recording of this talk?

Upgrading to WebDriverIO 5

A few weeks ago I finished upgrading our implementation of WebDriverIO from version 4 to version 5. The impetus for the upgrade was an announcement by the WebDriverIO twitter account of a new beta version 6 to be quickly followed by a finished version (it’s already here). One thing was clear: you have to be on v5 to go to v6 and each new subsequent version would only be supported for a year. Time to upgrade!

I’d been on version 4 since I originally deployed WebDriverIO in mid 2018. I knew version 5 was out but I had no immediate plans to upgrade given all of the warnings around breaking changes.

Preparation

This isn’t to say I wasn’t preparing myself. I created a JIRA ticket to outline what the work might look like. I was going through the TestAutomationU course on WebDriverIO which uses V5 and of course practicing in my own repo. With confirmation of the EOL of version 4 it was time to move on. 

I scoped out more of the work in JIRA (and also made a story to upgrade to v6). I bookmarked a few important articles including the WebDriverIO blog post announcing version 5 which highlights, among other things, specifically How to Upgrade. Then I made note of the methods that were either changing or being replaced for easier reference.

Upgrading to WebDriverIO 5

Finally it was time for the upgrade itself. The first question I needed to answer was logistical: how do I approach making the changes? Create a whole new repo, install the new packages and then move my tests over? Or just upgrade in place? Despite the daunting feeling I had, I figured it would be easier to upgrade in place and deal with the test failing as they came. This would introduce less changes than actually trying to move things over. Then it was time to follow the recommendations in the blog post:

  1. Remove node_module
  2. Remove all wdio-* packages from package.json
  3. Install the latest version of webdriverio: $ npm install [email protected]
    1. Note: if you did this today, you’d want to npm install [email protected] however I’d recommend going straight to v6 instead of going to v5 and then v6.
  4. Install the new wdio testrunner: $ npm install @wdio/cli --save-dev
  5. I have multiple configuration files and so I didn’t need to back them up. Rather I created a new one as part of the webdriverio configuration wizard. Then I eventually migrated / pruned the original ones until they were what I wanted.
  6. Rerun the configuration wizard: $ npx wdio config

All the Broken Things

Bam! WebDriverIO 5 deployed. Kind of. The easy part was done, next up was running each test one at a time, starting with the easiest / shortest tests. (I usually start with low hanging fruit so that I build momentum). When a test failed, I’d find out where and why (due to a rename or deprecation) and make a change. Rise and repeat for the whole test suite.

If this sounds simple, that’s because it was. All it took was time to remember where things were and why something was done a certain way and then make a change. 90% of the time this worked fine.

Other times it was more complicated. In at least one instance I ran into a breaking change, the functionality that was there before didn’t exist in the new version. I commented out 2 tests and moved on. (Speaking of which, I need to open a bug report about this).

Benefits

While it’s never fun to take something that’s working and break it just to upgrade, it does have some side benefits. Such as…. I fixed a few remaining bad ternary statements. Then cleaned up some unused libraries / plugins that I installed for some reason but can’t possibly remember anymore. Also refactored a bit of my code to make things simpler. When I was going through the TestAutomationU course I got to see how the author structured her tests. Now I have a new task to break apart my larger tests into smaller more defined tests with less assertions. 

All of this is to say, once you start making changes to improve one thing, it can snowball and lead to even more improvements. 

If you liked this article consider sharing it or buying me a coffee:

How I Became An Automation Engineer

Have you ever wondered what an Automation Engineer is or what they do? I’ve never found a great definition so I shared my experiences on How I Became an Automation Engineer and what that first year has looked like: the good, bad and the ok. I also talked a bit about the future of this role and the many challenges I see it facing (which might be a bit controversial).

Watch Now

Slides

References

Update:

Since the OnlineTestConf I’ve given this talk a few more times, the latest video you can find on my list of Publications. I’ve also written up some FAQs in case that helps!

It’s easier to write about tooling

It’s easier to write about tooling than it is to write about the decisions we took and models we made prior to choosing it. (The same is true for talking about tooling). I can write about a specific test I designed with WebDriverIO far easier than I can write about the strategy taken, oracles used or even the trade offs.

Aside from being a popular approach, there’s a lot of value in this directness. I’m able to succinctly communicate a specific problem and solution that might help someone else solve a similar problem.

The downside is when someone doesn’t understand this subtle communicative strategy and makes the wrong assumption(s) about the decision path. This seems to be a common pattern when talking with someone new to automation in testing: they just want the tool and none of the other fluff.

My solution for this problem is a bit of fishing: I will recommend a tool and then ask a lot of background questions even if that means I retract my initial recommendation.

Installing ChromeDriver on Windows

Installing on Windows 7 & 10:

The following are the best ways to install ChromeDriver on Windows so you can run Selenium:

  • The easiest way to install ChromeDriver is to use a package manager such as Chocolatey. You should really have a package manager if you don’t now:
    1. Install Chocolatey. I used the cmd.exe method and it worked really quickly.
    2. Install ChromeDriver choco install chromedriver
    3. Confirm it was installed by typing chromedriver —version into your command line and seeing that it returns a version number. If it errors, ChromeDriver wasn’t installed.
  • Or run Chrome & ChromeDriver in a container such as Docker. Simply download the combined container, start it and point your code at the right address. Here’s an example.

Which methods have you found the easiest or most success with? Which methods didn’t work for you? Please leave a comment below. 

Oh and if this article worked for you please consider sharing it or buying me coffee!

How To Run Your Selenium Tests with Headless Chrome

The Problem

If you want to run your tests headlessly on a Continuous Integration (CI) server you’ll quickly realize that you can’t with an out-of-the-box setup since there is no display output for the browser to launch in. You could use a third party library like Xvfb (tip 38) or PhantomJS (tip 46) but those can be hard to install and aren’t guaranteed to be supported in the long run (like PhantomJS).

A Solution

Enter Headless Chrome. Headless is simply a mode you can put Chrome into that allows it to run without displaying on screen but still gets you the same great results. This is a better option than using Chrome in a Headless manner such as in a docker container where the the container actually uses Xvfb.

Starting with Chrome 59 (Chrome 60 for Windows) we can simply pass Chrome a few configuration options to enable headless mode.

An Example in Ruby

Before we start make sure you’ve at least got the latest version of Chrome installed along with the latest version of ChromeDriver.

Let’s create a simple Selenium script (the example is posted below).

  1. We will pull in the requisite libraries and then create our setup method where we will pass Chrome our headless option as a command line argument. The first add_argument of ‘– headless’ allows us to run Chrome in headless mode. The second argument is, according to Google, temporarily required to work around a few known bugs. The third argument is optional but gives us the ability to debug our application in another browser if we need to (using localhost:9222).
  2. Now let’s finish our test by creating our teardown and run methods:

Here we are loading a page, asserting on the title (to make sure we are in the right place), and taking a screenshot to make sure our headless setup is working correctly. Here’s what our screenshot looks like:

headless

Expected Behavior

When we save our file and run it (e.g. ruby headless_chrome.rb) here is what will happen:

  1. An empty chrome browser instance will open
  2. Test runs and captures a screenshot
  3. Browser closes

Outro

Hopefully this tip has helped you get your tests running smoothly locally or on your CI Server. Happy Testing!

Additional References

This was originally written for and posted on the Elemental Selenium newsletter. I liked it so much I decided to cross post it with some updates.

Oh and if this article worked for you please consider sharing it or buying me coffee!

How To Run Your Selenium Tests Headlessly in Docker

The Old Way

It used to be that in order to get your Selenium tests running on a given machine you had to install each individual browser and then the browser drivers (for instance ChromeDriver for Chrome). Some of my most popular posts are about installing these drivers. However that’s all changed if we instead use containers. Simply download and run a single Docker container with both the the browser and the driver pre-installed.

Problem solved!

…Except now we’ll be running your tests headlessly which customers don’t do.

The Catch: Headless Browsers

In the context running a Selenium test headlessly in Chrome, it means Chrome launches and runs the test in the browser but you don’t see it because there is no GUI. Instead you get a command line to run and debug your tests. The downside is one customers don’t use your application in a headless state and two these tests are harder to debug.

The upside is we can manage the installation and running of the browsers and drivers in an easier way. We can also run our tests on devices that don’t have computer graphics like AWS EC2 instances (virtual machines), and CI/CD services like TravisCI and CircleCI to name a few.

Docker Desktop and Container Installation:

In this exercise we will download a single (standalone) Chrome browser with ChromeDriver already configured. Then we will simply run a test!

  • First thing we need to do is install Docker Desktop for our respective OS (Mac or Windows)
    • Once this is up and running we can download our container
    • (Note: this is a desktop app but can also be installed from the command line for future virtual machine use)
  • Second, since we already know what container we want, we simply need to copy and paste this command into our terminal docker pull selenium/standalone-chrome
    • This will download the Chrome container image
  • Third, we run is to start the container so Chrome is running
    • We can run docker ps at any time to see if any containers are running
    • We now run
      docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome
    • This sets the docker container to use the port 4444. We will use this later in our test file.

Running Our Tests: An Example

Now we’re ready to write a test using Selenium and our now running Docker container with Chrome in it. For fun I’m going to write this in Ruby but it should work for any language:

Expected Behavior

When we save our file and run it (e.g. ruby headless_chrome.rb) here is what will happen:

  • A chrome browser session will open within the docker container
  • Test runs and saves a screenshot
  • Browser closes
  • All we see is the result of the test

There you go! We managed to install a container with both Chrome and ChromeDriver installed on it and then wrote and ran a test to demonstrate it works. You’ve now run a Selenium Test Headlessly with Docker!

In the larger picture this means each time we need to install Selenium on a new computer (CI, EC2 instance, virtual machine, desktop) it’s simply about installing docker and downloading an image, spinning it up and connecting it to your test. After having done this half a dozen times I think it has huge advantages over the old ways (link to Installing ChromeDriver on macOS).

References:

  • Selenium Docker Hub for finding the approprite container
  • This is not the same as using Headless Chrome (which is a configuration option in modern versions of Chrome). To learn more about using Headless Chrome go here.

 

Oh and if this article worked for you please consider sharing it or buying me coffee!

8 Tools I use to Accelerate My Testing

Inspired by Justin Rorhman’s post of a similar name with a slight twist focusing on tools that generally help accelerate my testing.

As a test and quality specialist embedded in an engineering team I have a lot of work to do on any given day. Our engineering team’s goal is to ship quality software on a regular basis to deliver value to our customers. Naturally I rely heavily on a number of tools to help me better understand what is going on with the software or code under test and/or to accelerate what I’m able to do in a given amount of time.

Like Justin said in his article “Testing tools don’t make me a good tester, but they do make me a better tester. These tools help amplify things I need to do repeatedly or give me access to the product I wouldn’t have otherwise. They also shape the way I think about testing problems, for better and worse.”

In no particular order these tools are:

  1. GitHub. We use the GitHub workflow to deliver all of our code including application, library, and test automation. With each push to a particular branch our CI system builds the application and runs our unit and integration tests. With each Pull Request I can see all or just specific changes, code reviews and test run summaries. All this helps keep our build pipeline clean, makes it easier to identify potential issues quickly and helps me quickly build my test strategy.
  2. Docker. Each separate application of our system has been dockerized (aka runs in a container). This makes it incredibly easy to set new people up (a few hours of setup instead of a day) and gives us some powerful logging for each application. It also reduces conflicts when switching between branches where libraries or decencies have been changed.
  3. Heroku review apps. As I mentioned above with each push our application is compiled through our CI system. Once finished our CI system pushes that complied build to Heroku which automatically provisions a staging like environment accessible through its own url. (This feature is called a Heroku Review application). In about 5 minutes after a set of changes have been built our team now has a brand-new test environment ready for those hard to test local problems like testing on real mobile devices or a few special integrated services. Being able to build these new automatic quickly allows our developers to show off / get quicker input to the work they are doing.
  4. Browser DeveloperTools. I primarily use Chrome and Safari but the developer tools in each browser are priceless. From Inspecting requests and looking at their data, to setting or removing experiment variables, debugging problems, looking for problems, profiling, responsive design, layout, checking DOM elements, etc browser DeveloperTools really are the first place to start. It would be so much harder to learn what is going on without them.
  5. CircleCI. Great for keeping your deployment pipeline clean, running your build tests and ensuring those changes get where they are going each time. It doesn’t much matter what CI system you use it’s helping to accelerate something that would normally take away from developer or tester time.
  6. EverNote. Surprising or not, I’m always taking notes about any number of things like quick terminal commands, operations for creating data, migrating databases in heroku, setting up configurations, test ideas, questions for follow up, etc. Anything I might forget goes here and I can get to it anywhere, any place on any device.
  7. iTerm2. A command line replacement utility with Oh My Zsh installed that I’ve customized and continue to customize with shortcuts and aliases to make my repetitive tasks quicker. Also customized with visuals showing which repo I’m in, what branch I’m on and if there are uncommitted files, because I do enough on the command line.
  8. Automated UI acceptance tests. Running our acceptance tests for each staging deploy (or set of deploys) allows us time to do more deep exploration on the application as a whole with a reduced focus on those areas covered by the existing tests.

Part of the desire for writing this was to record what my work looks like now and then be able to compare it later-on in the future. However, in the time since I started writing this article I changed jobs and now the future is here. Well, kind of. My new team’s development stack is a lot more varied, which means so are the tools I’m using. I think it’s still best to wait for half a year or so before I write a follow up post on the tools accelerate that work. In the meantime, what tools do you use that accelerate your testing?

Good and Bad UI Test Automation explained – Inspired by Richard Bradshaw’s Tweets

Generally speaking there’s a scary trend with the influx of people interested in test automation where (seemingly) everyone wants to automate at the UI level. For example the phrase “Test automation” seems to be synonymous with UI automation which seems to mean using Selenium. To be fair there are numerous reasons for this. First, if you primarily focus on functional testing through the UI, this application of automation is the most relevant and straightforward to understand. Second, there are a lot of resources out there (books, blogs, classes) for Selenium. Third it is much easier to write some blackbox automation at the UI level then to develop competence at writing automation at the API or unit level. (You should still learn to do this lower.)

So Richard took to Twitter to explain his thoughts. Let’s take a look:

Before we go very far, let’s look at what Richard means by targeted tests.

Tests are “targeted” if they are designed to reveal a specific risk or failure (we might call this risk-based). If you are doing risk-based testing, your tests should ALWAYS be designed this way, regardless of how you intend to execute the tests (automated or not).

Seams are how you use the system’s implementation to help you build more streamlined and realiable automated tests. Seams can be can be APIs, mocks, stubs, or faked dependecies of your real system.

Want some examples of Seams? Checkout this blog post by Angie Jones.

Yes, most times we can use better designed tests! Want to test the various ways to get to a particular feature? Use PATH testing. Want to test the feature itself? Use functional or domain testing, etc. Some test techniques will be better at surfacing one type of problem and bad at surfacing another. It’s not always necessary to design your UI tests as long feature rich Scenarios.

When we talk of interfaces and automating at the lowest level it is usually a good time to mention this is what the Test Automation Pyramid tries to describe. There are lots of versions of this but I recommend Alan Page’s Test Automation Snowman.

If you can find the failure by writing a test lower in the stack (at the API level intead of UI level or at the unit level instead at the API level), the faster it runs and the faster you get feedback. If you do have to write the test at a higher level like the UI can you find some way to decrease the setup time / data of the test? (See above for Angie Jone’s blog post.) If you aren’t sure maybe thinking about whether you are testing the UI or testing through the UI will help?

Yes! Adding testability like IDs, classes, data-test attributes, setting up test data, mocking, etc. should be done as quickly and dependbly as possible. If you can’t do this on your own (or you have a different process for making changes) bring it up during sprint planning in order to get the development teams help behind building a solution.

This IS a thing. Selenium is great, it’s open source and the WebDriver protocol is the soon-to-be de facto standard for interacting with browsers. However there are lots of tools, some will use Selenium under the hood and others that are completely different (think Cypress.io). Depending on the problem you need to solve, you may find a specific tool much better than Selenium. GitHub is a really good resource for finding open source tools and when you find something that might work, give it a try.

If you want to write Good UI Test Automation your tests should be ideally suited for the UI (and not some other layer), targeted, reliable, use seams where possible andcover a specific risk. If you aren’t here yet work to refactor your test code until you can get there.

Did I miss anything about Richard’s tweet storm that was important?