If you want to run your tests headlessly on a Continuous Integration (CI) server you’ll quickly realize that you can’t with an out-of-the-box setup since there is no display output for the browser to launch in. You could use a third party library like Xvfb (tip 38) or PhantomJS (tip 46) but those can be hard to install and aren’t guaranteed to be supported in the long run (like PhantomJS).
Enter Headless Chrome. Headless is simply a mode you can put Chrome into that allows it to run without displaying on screen but still gets you the same great results. This is a better option than using Chrome in a Headless manner such as in a docker container where the the container actually uses Xvfb.
Starting with Chrome 59 (Chrome 60 for Windows) we can simply pass Chrome a few configuration options to enable headless mode.
An Example in Ruby
Before we start make sure you’ve at least got the latest version of Chrome installed along with the latest version of ChromeDriver.
Let’s create a simple Selenium script (the example is posted below).
We will pull in the requisite libraries and then create our setup method where we will pass Chrome our headless option as a command line argument. The first add_argument of ‘– headless’ allows us to run Chrome in headless mode. The second argument is, according to Google, temporarily required to work around a few known bugs. The third argument is optional but gives us the ability to debug our application in another browser if we need to (using localhost:9222).
Now let’s finish our test by creating our teardown and run methods:
Here we are loading a page, asserting on the title (to make sure we are in the right place), and taking a screenshot to make sure our headless setup is working correctly. Here’s what our screenshot looks like:
When we save our file and run it (e.g. ruby headless_chrome.rb) here is what will happen:
An empty chrome browser instance will open
Test runs and captures a screenshot
Hopefully this tip has helped you get your tests running smoothly locally or on your CI Server. Happy Testing!
It used to be that in order to get your Selenium tests running on a given machine you had to install each individual browser and then the browser drivers (for instance ChromeDriver for Chrome). Some of my most popular posts are about installing these drivers. However that’s all changed if we instead use containers. Simply download and run a single Docker container with both the the browser and the driver pre-installed.
…Except now we’ll be running your tests headlessly which customers don’t do.
The Catch: Headless Browsers
In the context running a Selenium test headlessly in Chrome, it means Chrome launches and runs the test in the browser but you don’t see it because there is no GUI. Instead you get a command line to run and debug your tests. The downside is one customers don’t use your application in a headless state and two these tests are harder to debug.
The upside is we can manage the installation and running of the browsers and drivers in an easier way. We can also run our tests on devices that don’t have computer graphics like AWS EC2 instances (virtual machines), and CI/CD services like TravisCI and CircleCI to name a few.
Docker Desktop and Container Installation:
In this exercise we will download a single (standalone) Chrome browser with ChromeDriver already configured. Then we will simply run a test!
Once this is up and running we can download our container
(Note: this is a desktop app but can also be installed from the command line for future virtual machine use)
Second, since we already know what container we want, we simply need to copy and paste this command into our terminal docker pull selenium/standalone-chrome
This will download the Chrome container image
Third, we run is to start the container so Chrome is running
We can run docker ps at any time to see if any containers are running
We now run docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome
This sets the docker container to use the port 4444. We will use this later in our test file.
Running Our Tests: An Example
Now we’re ready to write a test using Selenium and our now running Docker container with Chrome in it. For fun I’m going to write this in Ruby but it should work for any language:
When we save our file and run it (e.g. ruby headless_chrome.rb) here is what will happen:
A chrome browser session will open within the docker container
Test runs and saves a screenshot
All we see is the result of the test
There you go! We managed to install a container with both Chrome and ChromeDriver installed on it and then wrote and ran a test to demonstrate it works. You’ve now run a Selenium Test Headlessly with Docker!
In the larger picture this means each time we need to install Selenium on a new computer (CI, EC2 instance, virtual machine, desktop) it’s simply about installing docker and downloading an image, spinning it up and connecting it to your test. After having done this half a dozen times I think it has huge advantages over the old ways (link to Installing ChromeDriver on macOS).
Have you ever wanted to learn a little bit about Selenium WebDriver but didn’t know where to start?
Turns out there are some good tips / tutorials online for practicing writing Selenium in Ruby. One of those is a newsletter called Elemental Selenium that has something like 70 tips. You can sign up for the newsletter if you want but what I found valuable was to look at several of these tips, write them out (don’t copy + paste) and make sure you understand what they do. Turns out when you do this and you commit them to a repo, you can reference back to them when you come across similar problems in the future.
Simply stated [highlight]the goal is to:[/highlight]
Read through the Elemental Selenium tips and then write (don’t copy + paste) the code yourself.
Try running the tests locally and see how things work.
Once you’ve written a few tests, refactor those example tests so they become more DRY (don’t repeat yourself). Create page objects.
Commit these to your own repo on GitHub.
Putting your code on GitHub will have the benefit of showing you can write (at least) basic selenium automation. Although this code may not be your “best foot forward” given how new you’ll be, it is a starting point. As you learn more and make improvements, your code will reflect this.
These tips are (hopefully) grouped correctly but within each group there may be some variance. See if you can do one or two per day (some will be easier than others). If you see something interesting and want to jump to it immediately, go for it.
GeckoDriver is the library you need to download to be able to use Selenium WebDriver with Firefox. These are the Selenium Bindings.
Marionette is the protocol which Firefox uses to communicate with GeckoDriver. Installed by default with Firefox.
FirefoxDriver is the former name of GeckoDriver.
Ways to install GeckoDriver:
The easiest way to install GeckoDriver is to use a package manager like brew or npm such as npm install geckodriver. This method requires you some package manager installed but you probably should anyways.
Run Firefox and GeckoDriver in a container using Docker. Simply download the combined container, start it and point your code at the right address. I’ve written about how to do this using Chrome, should be very similar to do Firefox.
Early last month I was trying to update some gems on my MBP and ran into an SSL error:
ERROR: While executing gem ... (Gem::RemoteFetcher::FetchError)
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
In typical fashion I ignored the error and tried updating RubyGems (the package manager). That failed with the same SSL message. Additionally we run our automated tests as part of our continuous build process (which also uses RubyGems) and eventually this SSL problem resulted in TeamCity build errors:
Upon further debugging I found an error similar to the above error:
ERROR: Could not find a valid gem 'bundler' (>= 0), here is why: [22:36:17][Step 1/3] Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://api.rubygems.org/specs.4.8.gz) [22:36:17][Step 1/3] Process exited with code 2
Apparently RubyGems updated their SSL certificates which broke any and all gem updates from then on out. Luckily I found a thread (someone filed a bug) on RubyGem’s GitHub repo.
The solution however was a bit different depending for Windows (TeamCity) and MacOS (MBP). On my Mac I was able to use RVM to update the SSL certs using these commands (line 1 & 3):
Apparently on Windows you can’t just update the SSL certs; instead I had to re-install RubyGems. Simply re-download the latest version of RubyGems, go into the folder (for me it was installed at C:\RubyGems-2.6.3) and then run ‘setup.rb’. That was it!
I can understand the need to upgrade SSL certificates; it’s just a pain. Hopefully this makes it easier for others.
Occasionally something will change in our system under test that breaks a Selenium test (or two). Most of the time we can walk through the failure, make some tweaks and run it again – repeating the process until it passes. Depending on how long it’s been since we last worked with the code, or how deeply buried the code is, it may not be enough to fix the test and we may have to tackle one or more of the underlying methods we used to build the code.
In these situations it can be helpful to debug our tests using an interactive prompt or REPL. In the case of Ruby we can use irb or Interactive Ruby to manually step through the Selenium actions, watching and learning. Here’s the general format for working with Selenium in irb; it’s very similar to how we code it:
Launch interactive ruby: irb
First, import the Selenium library: require 'selenium-webdriver'
Second, create an instance of Selenium and launch the browser: driver = Selenium::WebDriver.for :firefox
Third, start entering your Selenium commands: driver.get 'http://www.google.com'
Here’s a simple example using irb:
In this example I’m bringing up Chrome as my browser and navigating to Google. I’m finding the search query box, typing my last name ‘Kenst’ and clicking enter. Thus an example of searching Google for my last name!
Technically once the browser is up we can navigate to whatever page we need without typing ALL of the individual commands. This is really valuable in those instances when you need to login, then navigate several pages before getting to the place you can debug. In other words do all the setup outside of irb, directly in the browser. Once you are in the proper location step through your code one command at a time (and lines with => show the responses to our commands, if any). Those responses will help us debug our tests by confirming what elements Selenium is picking up and what our methods are returning (if anything).
tl;dr If you’ve ever wanted to learn Selenium but didn’t know where to start, The Selenium GuideBook is the place (doesn’t matter which edition you use, it’ll be good).
The challenge of trying to learn Selenium WebDriver (simply referred to as Selenium) or any relatively new technology is that there is either too much disparate information (on the web), the material is out of date (many of the books I found) or of poor quality. With Selenium too many of the tutorials available to beginners had you using the Selenium IDE, which is a really poor option for writing maintainable and reusable test code. (See my previous post’s video for more on this.) I’ve walked out on conference workshops that sought to teach people to use the Selenium IDE to start their automation efforts. It wasn’t for me. I was going to do it right, I just had to figure out what that meant and where to start.
From the start I knew I wanted to learn about GUI test automation and more specifically Selenium WebDriver. I had tried WATIR (Web Application Testing in Ruby) and a few other tools but Selenium was open source and quickly becoming the go-to standard for automating human interaction with web browsers. It was and is the only choice.
Naturally I went searching the web for some tutorial or examples when I stumbled across several tutorials including Sauce Lab’s boot camp written by someone named Dave Haeffner. After struggling through the Bootcamp series (and finding some bugs in the process) I found Dave also produced a tip series called Elemental Selenium. I signed up for the Ruby weekly newsletter tips and went through many of the tips. Satisfied that Dave was worth learning from (good quality, relevant code examples) I decided it was time to try his book The Selenium GuideBook. I knew going into it, I was going to be the person maintaining the test suite and since I was more or less comfortable with Ruby I was happy The Selenium GuideBook came in that language!
The package itself contains a lot of great information and a number of materials:
The Selenium GuideBook; the Ruby edition is roughly 100 pages
Ruby code examples broken out by chapter
Elemental Selenium Tips eBook in Ruby
The Automated Visual Testing Guidebook
The first time I went through the book and code examples, it seemed redundant having different code for each chapter and example. It was only after I had gone through the chapters and examples for a second time, trying to apply and understand the differences that I began to understand the relative importance of seeing the code change chapter by chapter, example by example. The code examples all target an open source application Dave created called The Internet. It seems simple enough but many of the books and materials I went through either tried using Google or some badly written / hurried example.
Despite being less than 100 pages the Selenium GuideBook covers:
Defining a test strategy for your automation goals
Using page objects and base page objects to keep code clean
Locator strategies and tools
Relying on good locators seemed like a smart way to design tests. I wanted to avoid any record and playback tools and the poor locator strategies often employed.
How to write good acceptance tests
Writing Re-usable test code
Running on browsers locally and then in the cloud
Running tests in parallel
Taking your newly built tests, tagging them and getting them loaded into a CI server.
The whole package literally. In the preface Dave says the book is not full and comprehensive, it’s more of a distilled and actionable guide. The book really shows you how to start thinking about and building a test suite. It’s up to the user to take what they learned here and apply it to their application. That’s the fun part of the Elemental Selenium tips as well.
Applying the Book and Examples
After I had gone through all of the chapters and examples once, I went back through the relevant to me chapters and examples doing the following:
Start with some very simple login tests.
The book starts out this way as well. Writing tests that are not DRY or re-usable. but eventually get that way.
Continuing through the code examples, getting a little more complicated and applying it to my own application.
As I built out tests and start to see commonly repeated patterns, abstract out repeated code into relevant page objects. Eventually getting to a base page object.
In hindsight the hardest part of applying the book was trying to understand and apply a locator strategy within our application. While The Internet target application is great, it’s also a bit simplistic. Good for teaching, hard for bridging the sample application to the target application. Our single page .NET application was far more complicated and it took several attempts before I understood how my own strategy would work.
The transfer problem is always difficult. I mean how do you take what you learned and apply it to a slightly more sophisticated problem? It’s a difficult problem, not really a criticism of this book. It’s worth noting that whenever I had questions about what was written in the book, found a bug or two, or got stuck I could email Dave and within a week or so get a helpful response back.
With the lessons in this book and the Elemental Selenium Tips I was, through some focused time and lots of iterations, able to get a fairly good set of automated acceptance tests running against our application.
In other words, I highly recommend you buy the book. It’s slightly more expensive than similar books about Selenium but it’s far more effective. You are also directly supporting the author and with free updates and email tech support I think its well worth the cost.
Don’t believe me? Watch this video of Dave giving an overview of the GuideBook:
At work we use TeamCity as our CI service to automate the build and deployment of our software to a number of pre-production environments for testing and evaluation. Since we’re already bottling up all the build and deployment steps for our software, I figured we could piggy back on this process and kick off a simple login test. It seems faster and easier to have an automated test tell us this, than to wait until someone stumbles across it. After all who cares if a server has the latest code if you can’t login to use it?
Note: I’m calling the test that attempts to login to our system a sanity test. It could just as easily described it as a smoke test.
The strategy looked something like:
Make sure tests are segmented (at least one ‘sanity’ test)
Hook up tests to Jenkins as a proof of concept
Once the configuration works in Jenkins (and I’ve figured out any additional pre-reqs), reconfigure tests in TeamCity to run “sanity tests” (which is a tag)
If sanity tests prove stable, add additional tests or segments
Segmenting tests is a great way to run certain parts of the test suite at a time. Initially this would be a single login test since logging in is a pre-cursor to anything else we’d want to do in the app. For our test framework RSpec this was done by specifying a ‘sanity’ tag.
There didn’t appear to be any guidelines or instructions on the interwebs on how you might configure TeamCity to run RSpec tests but I found them for another CI, Jenkins. Unlike TeamCity, Jenkins is super easy to set up and configure: download the war file from the Jenkins homepage, launch it from the terminal and create a job! Our test code is written in ruby which means I can kick off our tests from the command line using a rake file. Once a job was created and the command line details were properly sorted, I was running tests with the click of a button! (Reporting and other improvements took a little longer). Note, we don’t use any special RSpec runner for this, just a regular old command line, although we do have a Gemfile with all relevant depencies listed.
Since I couldn’t find any guidelines on how to configure TeamCity to run RSpec accpetance tests, I’m hoping this helps. We already had the server running so this assumes all you need to do is add your tests to an existing service. After some trial and error here’s how we got it to work:
Added version control settings for the automation repo
Within the build configuration added 3 steps:
Install Bundler. This is a command line step that runs a custom script when the previous step finishes. Basically handles the configuration information for Sauce Labs (our Selenium grid provider) and the first pre-req.
Bundle Install. Also a command line step running a custom script. Second pre-req.
Run Tests. Final command line step using rake and my test configuration settings to run the actual tests
Added a build trigger to launch the sanity tests after a successful completion of the previous step (deploy to server)
After this was all put in, I manually triggered the configuration execution to see how well the process worked. There were quite a few hiccups along the way. One of the more interesting problems was finding out the TeamCity agent machine had outdated versions of Ruby and Ruby gems. The version of Ruby gems was so out of date it couldn’t be upgraded, it had to be re-installed which is never much fun on an RDP session.
Once the execution went well I triggered a failure. When the tests fail they print “tests failed” to the build log. Unfortunately the server didn’t seem to understand when a failure occurred so I went back and added a specific “failure condition” (another configuration option) looking for the word “tests failed” which, if found, would mark the test as a failure. Simple enough!
We’ve been running this sanity test for a few months now and it’s quite valuable to know when an environment is in an unusable state and yet I think visibility is still a challenge. Although the failures report directly into a slack channel I’m not sure how quickly the failures are noticed and/or if the information reported in the failed test is useful.
A few articles I’ve read suggest using the CI server to launch integration tests instead of the UI level acceptance tests we are running. I think what we are doing is valuable and I’d like to expand it. I wonder what additional sanity or segments of tests do we add to this process? Are there more or better ways to do what we’re doing now? Please share your experiences!
The main product I test was designed to follow a responsive web design layout so it could theoretically be used on anything from desktop computers to tablets and smartphones. Practically speaking this means different viewable window sizes (viewport sizes) will result in the browser placing elements of our application in different locations on the screen. When running my selenium acceptance tests I wanted to be able to specify different viewport sizes both locally and remotely on Sauce Labs. While the sizes may not make a difference to Selenium they give me another variable to specify if I so choose to do responsive testing. The examples I found weren’t very helpful so I decided to make my own for both.
Resizing your window. Locally the browser can be resized to a specific width and height by using the resize_to() command. For a window size of 1280×1024 the line of code we are looking for is:
If you don’t use a helper spec you can include this code in your setup method or right after you call WebDriver.
Setting screen resolution. Resizing your window works great locally but what if you want to run your tests remotely at Sauce Labs? How do we ensure our screen resolution is large enough to support a larger window size? Luckily Sauce Labs opens their browsers to the maximum window size so all we have to do is set the screen resolution.
If we go back to the example-selenium repo you’ll see I’m actually using caps[“screenResolution”] = ENV[‘resolution’] in the above spec_helper at line 11.
I’m setting a global variable for screen resolution so I can update it in the config_cloud file as I might update other global settings like operating system or browser version. This is important because in some cases, I may have to either adjust the resolution size or in the case of Safari, actually comment it out. For some reason Sauce Labs doesn’t have many resolutions options for Mac OS X, which is a bit annoying. The latest versions of OS X don’t even support resolutions of 1280×1024.
I’ve been building a GUI acceptance test automation suite locally in Ruby using the RSpec framework. When it was time to get the tests running remotely on Sauce Labs, I ran into the following error:
RSpec::Core::ExampleGroup::WrongScopeError: `example` is not available from within an example (e.g. an `it` block) or from constructs that run in the scope of an example (e.g. `before`, `let`, etc). It is only available on an example group (e.g. a `describe` or `context` block).
occurred at /usr/local/rvm/gems/ruby-2.1.2/gems/rspec-core-3.2.2/lib/rspec/core/example_group.rb:642:in `method_missing'
It took a few minutes debugging before I spotted the error: