For the last few months I’ve been using a no-code UI test automation service called Reflect.run to build out some UI tests (scenarios and such) with the goal of evaluating how well it works in in terms of feedback (and value) as part of our build process. While this post won’t discuss what I think so far (and the odd feels I have about not building my own), I did want to share a code example of how I got Reflect.run builds to run in our CI pipeline.
Our current CI tool is called CodeFresh. Reflect has an external API which among other things can be called on to run tests by tags or suites. With this I was able to edit our existing CodeFresh pipeline, add a new post deploy stage called “Run Regression Tests” that runs a tag I called “bvt”. (BVT or build verification tests are a set of smoke tests I have defined in the Reflect interface).
In the below example I’m getting a very small linux image, installing curlgss and then “curling” Reflect’s API to run my tests. (If you don’t install curl first, you can’t make the curl call.) Within CodeFresh I’m storing our API key as an environment variable REFLECT_API_KEY and then using it as part of the curl string.
That’s all there is to getting Reflect.run‘s tests to run as part of a CodeFresh build. Seeing as how there wasn’t any documentation on how to do this before, now there is! (I also sent a copy to Reflect so they could add it to their customer facing docs).
In March I went through the process of upgrading to WebDriver v5. Last month I took the next step by upgrading our deployment to version 6 so we’d be current. I learned quite a bit from that first upgrade which made this upgrade a whole lot easier.
Here’s what I did to upgrade to WebDriverIO 6:
Checked the Docs. I scanned through the change log sections on breaking changes. Maybe I’m boring but nothing in the docs breaking changes notes looked to impact our setup! 👏
npm oudated . This is one of my favorite (and mostly unused) npm commands. It told me specifically what packages I had that were outdated. Hint: it was all of them. 🤯
Upgrade the easy stuff.
Based on what npm returned, I began by updating my package.json file for 3rd party libraries. Libraries such as prettier, chromedriver, moment, etc. that all played a part in my solution but didn’t deal directly with my tests. Then I npm install the latest changes.
Run the whole test suite looking for failures.
Upgrade the harder stuff.
Same thing as step 3, updated the package.json versions to be the “latest” and then npm install.
Another way to do this is to remove the node_modules folder and re-install each package based on the install instructions.
Run the whole test suite… until nothing is broken!
This upgrade wasn’t as daunting as I initially feared. With upgrades there’s always some level of concern you are going to “ruin a good thing”. Once I moved past that concern, there wasn’t much effort involved in getting things to work. Lots of credit goes to the WebDriverIO team for making the process more reasonable and straightforward, which of course they stated in their announcement:
This major update is much more reasonable and contains subtle changes that will help the project further grow while remaining performant at the same time.
I’ve been pretty happy thus far using WebDriverIO and I’m excited to see where things go!
I’ve given the talk on How I Became an Automation Engineer a few times. Each time I’ve gotten good feedback and a lot of questions from engaged participants. With each repeated question, I try to update my talk to address those points for future audiences. However often I do this some repeated questions still arise. I’ve taken this challenge as both an excuse to write out my answers and try out a FAQ schema block!
Frequently Asked Questions
How do you decide the scripting or programming language to use in your testing? Do you typically use the same language the app is built in?
Ultimately your goal is to tackle the task at hand. Use the tools and languages that make sense.
Is (insert name of your current tool) sufficient enough to perform all types of automation testing?
No, probably not. Hopefully you work for an organization that wants to use the right tool for the right job.
Part of the process for learning about tools is we figure out what they are and aren’t good for. No one tool is good for all jobs. The same goes for the person using the tool. If you only know one tool you’ll try to use it for every problem. The more tools we learn, the more likely we will use the right tool for the right job. This also makes us more valuable in the market for new jobs.
What are your thoughts on using frameworks with record features to start learning automation?
It depends on your needs and your skills.
In the past, Record and Playback features were sold as Snake Oil. They’d fix all your problems. But they usually were much good beyond proof of concepts.
Today, more and more services are cropping up with codeless / record and playback capabilities. If these meet the needs of your project and/or company, then use them. One challenge for the individuals learning and using these these tools is they might not benefit your (market) value in the long term.
Sometimes the team or project doesn’t support automation either for budget or time concerns. How do you tackle that?
Test Automation is there to provide feedback, but it costs money and takes time to build and maintain. So if you are on a short term project then it might not be worth the time an effort to do certain kinds of test automation.
What kinds make sense? Well it depends on the project. It might be worth it to do unit tests but not much else. Or it might be worth it to build some one-off probabilistic automated tests to help you discover problems because it doesn’t have to be maintained. Remember the point of automation is to reduce our time or extend our capabilities but all that must be done within the context of your project.
Do you recommend learning to test and automate at the API level? Is Postman adequate for this?
Yes! In general I recommend automating tests at the unit level first, then at the service or API level and lastly at the UI level. Learning to explore APIs is a big part of this and eventually once you explore them, you can create automated tests.
Postman is a good tool for exploring APIs (there are probably others). It might also make sense for you to write some automated tests in Postman but that really depends on your goals and/or if there are better places to do it. This depends on your overall strategy.
According to your last slide, the future of the Automation Engineer role is uncertain. Do you think it’s better to move to the developer role or there is still hope that automation roles will be still exist?
We know that in high performing teams developers own the automated tests and testers are allowed to explore for risks. So there’s a high probability in companies looking to build high performing teams that Automation Engineering roles will disappear or not exist. There’s also a high probability of developer roles being needed in the future. We just don’t know when any of this will occur. If you are going to move, it makes sense to survey the landscape and see what the options are, see what your skill levels are and be practical.
What is the name of the book about you recommend / quote about high performing teams handling test automation?
A few weeks ago I finished upgrading our implementation of WebDriverIO from version 4 to version 5. The impetus for the upgrade was an announcement by the WebDriverIO twitter account of a new beta version 6 to be quickly followed by a finished version (it’s already here). One thing was clear: you have to be on v5 to go to v6 and each new subsequent version would only be supported for a year. Time to upgrade!
I’d been on version 4 since I originally deployed WebDriverIO in mid 2018. I knew version 5 was out but I had no immediate plans to upgrade given all of the warnings around breaking changes.
This isn’t to say I wasn’t preparing myself. I created a JIRA ticket to outline what the work might look like. I was going through the TestAutomationU course on WebDriverIO which uses V5 and of course practicing in my own repo. With confirmation of the EOL of version 4 it was time to move on.
I scoped out more of the work in JIRA (and also made a story to upgrade to v6). I bookmarked a few important articles including the WebDriverIO blog post announcing version 5 which highlights, among other things, specifically How to Upgrade. Then I made note of the methods that were either changing or being replaced for easier reference.
Upgrading to WebDriverIO 5
Finally it was time for the upgrade itself. The first question I needed to answer was logistical: how do I approach making the changes? Create a whole new repo, install the new packages and then move my tests over? Or just upgrade in place? Despite the daunting feeling I had, I figured it would be easier to upgrade in place and deal with the test failing as they came. This would introduce less changes than actually trying to move things over. Then it was time to follow the recommendations in the blog post:
Note: if you did this today, you’d want to npm install [email protected] however I’d recommend going straight to v6 instead of going to v5 and then v6.
Install the new wdio testrunner: $ npm install @wdio/cli --save-dev
I have multiple configuration files and so I didn’t need to back them up. Rather I created a new one as part of the webdriverio configuration wizard. Then I eventually migrated / pruned the original ones until they were what I wanted.
Rerun the configuration wizard: $ npx wdio config
All the Broken Things
Bam! WebDriverIO 5 deployed. Kind of. The easy part was done, next up was running each test one at a time, starting with the easiest / shortest tests. (I usually start with low hanging fruit so that I build momentum). When a test failed, I’d find out where and why (due to a rename or deprecation) and make a change. Rise and repeat for the whole test suite.
If this sounds simple, that’s because it was. All it took was time to remember where things were and why something was done a certain way and then make a change. 90% of the time this worked fine.
Other times it was more complicated. In at least one instance I ran into a breaking change, the functionality that was there before didn’t exist in the new version. I commented out 2 tests and moved on. (Speaking of which, I need to open a bug report about this).
While it’s never fun to take something that’s working and break it just to upgrade, it does have some side benefits. Such as…. I fixed a few remaining bad ternary statements. Then cleaned up some unused libraries / plugins that I installed for some reason but can’t possibly remember anymore. Also refactored a bit of my code to make things simpler. When I was going through the TestAutomationU course I got to see how the author structured her tests. Now I have a new task to break apart my larger tests into smaller more defined tests with less assertions.
All of this is to say, once you start making changes to improve one thing, it can snowball and lead to even more improvements.
Have you ever wondered what an Automation Engineer is or what they do? I’ve never found a great definition so I shared my experiences on How I Became an Automation Engineer and what that first year has looked like: the good, bad and the ok. I also talked a bit about the future of this role and the many challenges I see it facing (which might be a bit controversial).
It’s easier to write about tooling than it is to write about the decisions we took and models we made prior to choosing it. (The same is true for talking about tooling). I can write about a specific test I designed with WebDriverIO far easier than I can write about the strategy taken, oracles used or even the trade offs.
Aside from being a popular approach, there’s a lot of value in this directness. I’m able to succinctly communicate a specific problem and solution that might help someone else solve a similar problem.
The downside is when someone doesn’t understand this subtle communicative strategy and makes the wrong assumption(s) about the decision path. This seems to be a common pattern when talking with someone new to automation in testing: they just want the tool and none of the other fluff.
My solution for this problem is a bit of fishing: I will recommend a tool and then ask a lot of background questions even if that means I retract my initial recommendation.
If you want to run your tests headlessly on a Continuous Integration (CI) server you’ll quickly realize that you can’t with an out-of-the-box setup since there is no display output for the browser to launch in. You could use a third party library like Xvfb (tip 38) or PhantomJS (tip 46) but those can be hard to install and aren’t guaranteed to be supported in the long run (like PhantomJS).
Enter Headless Chrome. Headless is simply a mode you can put Chrome into that allows it to run without displaying on screen but still gets you the same great results. This is a better option than using Chrome in a Headless manner such as in a docker container where the the container actually uses Xvfb.
Starting with Chrome 59 (Chrome 60 for Windows) we can simply pass Chrome a few configuration options to enable headless mode.
An Example in Ruby
Before we start make sure you’ve at least got the latest version of Chrome installed along with the latest version of ChromeDriver.
Let’s create a simple Selenium script (the example is posted below).
We will pull in the requisite libraries and then create our setup method where we will pass Chrome our headless option as a command line argument. The first add_argument of ‘– headless’ allows us to run Chrome in headless mode. The second argument is, according to Google, temporarily required to work around a few known bugs. The third argument is optional but gives us the ability to debug our application in another browser if we need to (using localhost:9222).
Now let’s finish our test by creating our teardown and run methods:
Here we are loading a page, asserting on the title (to make sure we are in the right place), and taking a screenshot to make sure our headless setup is working correctly. Here’s what our screenshot looks like:
When we save our file and run it (e.g. ruby headless_chrome.rb) here is what will happen:
An empty chrome browser instance will open
Test runs and captures a screenshot
Hopefully this tip has helped you get your tests running smoothly locally or on your CI Server. Happy Testing!
It used to be that in order to get your Selenium tests running on a given machine you had to install each individual browser and then the browser drivers (for instance ChromeDriver for Chrome). Some of my most popular posts are about installing these drivers. However that’s all changed if we instead use containers. Simply download and run a single Docker container with both the the browser and the driver pre-installed.
…Except now we’ll be running your tests headlessly which customers don’t do.
The Catch: Headless Browsers
In the context running a Selenium test headlessly in Chrome, it means Chrome launches and runs the test in the browser but you don’t see it because there is no GUI. Instead you get a command line to run and debug your tests. The downside is one customers don’t use your application in a headless state and two these tests are harder to debug.
The upside is we can manage the installation and running of the browsers and drivers in an easier way. We can also run our tests on devices that don’t have computer graphics like AWS EC2 instances (virtual machines), and CI/CD services like TravisCI and CircleCI to name a few.
Docker Desktop and Container Installation:
In this exercise we will download a single (standalone) Chrome browser with ChromeDriver already configured. Then we will simply run a test!
Once this is up and running we can download our container
(Note: this is a desktop app but can also be installed from the command line for future virtual machine use)
Second, since we already know what container we want, we simply need to copy and paste this command into our terminal docker pull selenium/standalone-chrome
This will download the Chrome container image
Third, we run is to start the container so Chrome is running
We can run docker ps at any time to see if any containers are running
We now run docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome
This sets the docker container to use the port 4444. We will use this later in our test file.
Running Our Tests: An Example
Now we’re ready to write a test using Selenium and our now running Docker container with Chrome in it. For fun I’m going to write this in Ruby but it should work for any language:
When we save our file and run it (e.g. ruby headless_chrome.rb) here is what will happen:
A chrome browser session will open within the docker container
Test runs and saves a screenshot
All we see is the result of the test
There you go! We managed to install a container with both Chrome and ChromeDriver installed on it and then wrote and ran a test to demonstrate it works. You’ve now run a Selenium Test Headlessly with Docker!
In the larger picture this means each time we need to install Selenium on a new computer (CI, EC2 instance, virtual machine, desktop) it’s simply about installing docker and downloading an image, spinning it up and connecting it to your test. After having done this half a dozen times I think it has huge advantages over the old ways (link to Installing ChromeDriver on macOS).
Inspired by Justin Rorhman’s post of a similar name with a slight twist focusing on tools that generally help accelerate my testing.
As a test and quality specialist embedded in an engineering team I have a lot of work to do on any given day. Our engineering team’s goal is to ship quality software on a regular basis to deliver value to our customers. Naturally I rely heavily on a number of tools to help me better understand what is going on with the software or code under test and/or to accelerate what I’m able to do in a given amount of time.
Like Justin said in his article “Testing tools don’t make me a good tester, but they do make me a better tester. These tools help amplify things I need to do repeatedly or give me access to the product I wouldn’t have otherwise. They also shape the way I think about testing problems, for better and worse.”
In no particular order these tools are:
GitHub. We use the GitHub workflow to deliver all of our code including application, library, and test automation. With each push to a particular branch our CI system builds the application and runs our unit and integration tests. With each Pull Request I can see all or just specific changes, code reviews and test run summaries. All this helps keep our build pipeline clean, makes it easier to identify potential issues quickly and helps me quickly build my test strategy.
Docker. Each separate application of our system has been dockerized (aka runs in a container). This makes it incredibly easy to set new people up (a few hours of setup instead of a day) and gives us some powerful logging for each application. It also reduces conflicts when switching between branches where libraries or decencies have been changed.
Heroku review apps. As I mentioned above with each push our application is compiled through our CI system. Once finished our CI system pushes that complied build to Heroku which automatically provisions a staging like environment accessible through its own url. (This feature is called a Heroku Review application). In about 5 minutes after a set of changes have been built our team now has a brand-new test environment ready for those hard to test local problems like testing on real mobile devices or a few special integrated services. Being able to build these new automatic quickly allows our developers to show off / get quicker input to the work they are doing.
Browser DeveloperTools. I primarily use Chrome and Safari but the developer tools in each browser are priceless. From Inspecting requests and looking at their data, to setting or removing experiment variables, debugging problems, looking for problems, profiling, responsive design, layout, checking DOM elements, etc browser DeveloperTools really are the first place to start. It would be so much harder to learn what is going on without them.
CircleCI. Great for keeping your deployment pipeline clean, running your build tests and ensuring those changes get where they are going each time. It doesn’t much matter what CI system you use it’s helping to accelerate something that would normally take away from developer or tester time.
EverNote. Surprising or not, I’m always taking notes about any number of things like quick terminal commands, operations for creating data, migrating databases in heroku, setting up configurations, test ideas, questions for follow up, etc. Anything I might forget goes here and I can get to it anywhere, any place on any device.
iTerm2. A command line replacement utility with Oh My Zsh installed that I’ve customized and continue to customize with shortcuts and aliases to make my repetitive tasks quicker. Also customized with visuals showing which repo I’m in, what branch I’m on and if there are uncommitted files, because I do enough on the command line.
Automated UI acceptance tests. Running our acceptance tests for each staging deploy (or set of deploys) allows us time to do more deep exploration on the application as a whole with a reduced focus on those areas covered by the existing tests.
Part of the desire for writing this was to record what my work looks like now and then be able to compare it later-on in the future. However, in the time since I started writing this article I changed jobs and now the future is here. Well, kind of. My new team’s development stack is a lot more varied, which means so are the tools I’m using. I think it’s still best to wait for half a year or so before I write a follow up post on the tools accelerate that work. In the meantime, what tools do you use that accelerate your testing?