How to export environments from Postman

Postman can export data, including collections and environments, to be used outside of Postman. This is especially important when using Newman (their command-line collection runner).

With the release of Postman v8.2 it is easier to find and export Postman collections but harder to find and export environments, hence this article.

(more…)

How to take a screenshot and record video in macOS

One of the nice things about macOS are the built-in tools. A recently improved upon built-in tool is called Screenshot which allows users to take a screenshot and record video. Taking screenshots has been around macOS forever, but taking video directly from OS shortcuts came along in macOS Mojave. The challenge is remembering the commands.

This tutorial is an attempt to capture the quick commands / shortcuts to make this process easy.

(more…)

How to debug problems on Mobile Safari

When you are developing and/or testing a web app on iOS, having the ability to inspect and understand what is happening in the browser is incredibly useful. For just this reason Mobile Safari can use the same developer tools that are built into Safari on macOS.

To start debugging you’ll need your iOS device, a mac and a lightning cable. Then start by making sure both devices are configured to allow this.

Configure devices:

  1. Launch Safari on your mac; open preferences and within the advanced tab click the checkbox for “Show Develop menu in menu bar”.
    1. Confirm, you should now see a menu called Develop in the top bar
  2. On your iOS device (iPhone or iPad) go to Settings > Safari > Advanced and slide the toggle on for Web Inspector.

Debug a site in Mobile Safari:

  1. Connect your iOS device (iPhone or iPad) to your mac
  2. On the device browse to your site in Mobile Safari
  3. On your mac’s Safari, under the Develop menu you should see an option for your iOS device (either called iPhone or iPad).
  4. In that sub-menu you should see the url of the website you want to debug. Click that url and it will open a new Safari window with the full Safari Dev Tools.
  5. Start your investigation!

Safari Dev Tools

A few weeks back I was testing changes to review submissions (buy a product, leave a review) and on Chrome desktop everything worked fine (including Chrome’s mobile emulator). However after testing on a real device, Safari didn’t behave the same way. Luckily we were able to debug using this same method and fix those remaining issues before any customers saw problems.

It just goes to show real device testing matters!

Oh and if this article worked for you please consider sharing it or buying me coffee!

8 Tools I use to Accelerate My Testing

Inspired by Justin Rorhman’s post of a similar name with a slight twist focusing on tools that generally help accelerate my testing.

As a test and quality specialist embedded in an engineering team I have a lot of work to do on any given day. Our engineering team’s goal is to ship quality software on a regular basis to deliver value to our customers. Naturally I rely heavily on a number of tools to help me better understand what is going on with the software or code under test and/or to accelerate what I’m able to do in a given amount of time.

Like Justin said in his article “Testing tools don’t make me a good tester, but they do make me a better tester. These tools help amplify things I need to do repeatedly or give me access to the product I wouldn’t have otherwise. They also shape the way I think about testing problems, for better and worse.”

In no particular order these tools are:

  1. GitHub. We use the GitHub workflow to deliver all of our code including application, library, and test automation. With each push to a particular branch our CI system builds the application and runs our unit and integration tests. With each Pull Request I can see all or just specific changes, code reviews and test run summaries. All this helps keep our build pipeline clean, makes it easier to identify potential issues quickly and helps me quickly build my test strategy.
  2. Docker. Each separate application of our system has been dockerized (aka runs in a container). This makes it incredibly easy to set new people up (a few hours of setup instead of a day) and gives us some powerful logging for each application. It also reduces conflicts when switching between branches where libraries or decencies have been changed.
  3. Heroku review apps. As I mentioned above with each push our application is compiled through our CI system. Once finished our CI system pushes that complied build to Heroku which automatically provisions a staging like environment accessible through its own url. (This feature is called a Heroku Review application). In about 5 minutes after a set of changes have been built our team now has a brand-new test environment ready for those hard to test local problems like testing on real mobile devices or a few special integrated services. Being able to build these new automatic quickly allows our developers to show off / get quicker input to the work they are doing.
  4. Browser DeveloperTools. I primarily use Chrome and Safari but the developer tools in each browser are priceless. From Inspecting requests and looking at their data, to setting or removing experiment variables, debugging problems, looking for problems, profiling, responsive design, layout, checking DOM elements, etc browser DeveloperTools really are the first place to start. It would be so much harder to learn what is going on without them.
  5. CircleCI. Great for keeping your deployment pipeline clean, running your build tests and ensuring those changes get where they are going each time. It doesn’t much matter what CI system you use it’s helping to accelerate something that would normally take away from developer or tester time.
  6. EverNote. Surprising or not, I’m always taking notes about any number of things like quick terminal commands, operations for creating data, migrating databases in heroku, setting up configurations, test ideas, questions for follow up, etc. Anything I might forget goes here and I can get to it anywhere, any place on any device.
  7. iTerm2. A command line replacement utility with Oh My Zsh installed that I’ve customized and continue to customize with shortcuts and aliases to make my repetitive tasks quicker. Also customized with visuals showing which repo I’m in, what branch I’m on and if there are uncommitted files, because I do enough on the command line.
  8. Automated UI acceptance tests. Running our acceptance tests for each staging deploy (or set of deploys) allows us time to do more deep exploration on the application as a whole with a reduced focus on those areas covered by the existing tests.

Part of the desire for writing this was to record what my work looks like now and then be able to compare it later-on in the future. However, in the time since I started writing this article I changed jobs and now the future is here. Well, kind of. My new team’s development stack is a lot more varied, which means so are the tools I’m using. I think it’s still best to wait for half a year or so before I write a follow up post on the tools accelerate that work. In the meantime, what tools do you use that accelerate your testing?

Lessons Learned from the Contributing to GitHub is For You Workshop

On April 11, 2018 I gave a workshop called Contributing to GitHub is For You is For You (abbreviated as C2GI4U) in Des Moines, IA to the Des Monies Area Quality Assurance Association (abbrevitated as DAQAA). Roughly 36 people signed up for the DAQAA event meetup and of those about 22 showed up. Scheduled for 3 hours (including a break in the middle for food) I set out to cover a number of topics around using Git + GitHub including:

  • Creating repositories
  • Branching
  • Submitting changes in Pull Requests
  • Merging, reverting, cloning, forking, and other common commands, etc.
  • Using git locally from the command line and GitHub remotely through the web

In a little less than 3 hours we were able to run through three quarters of the exercises, covered all of the basic git commands and many of the intricacies of using the Git + GitHub workflow. Everyone walked away knowing how to grab and contribute to a project in Github using Git in the command line. A few people were overwhelmed having to pick up terminal commands in addition to Git commands, but most were able to get through the exercises and commit jointly to the C2GI4U repository. By the end a few felt so empowered they were going to apply what they learned the next day at work. (This point made me feel quite well.)

Workshops (generally) are fun because we get to do hands-on technical things. They also bring up interesting challenges, some of which are expected and others not so much. For example I expected some people might have to learn to navigate the command line (or re-familiarize themselves). Although we split time between GitHub’s nice UI you can’t get away from the command line but with practice during the workshop many people became more comfortable. However there were a few things that I didn’t expect and hope to improve upon the next time:

  • Provide a list of git commands we will use alongside the exercises. I had a list of all the git commands we were using as well as a list of exercises BUT they were in separate places.
    • I included a few cheat sheets with a bunch of different git commands but these were too broad to teach from, especially for people brand new to git.
  • Include information about storing GitHub up credentials. Although this wasn’t a big issue (you get a prompt to login) I always forget this step since my credentials are saved in my terminal. Everyone ran into this problem.
  • Make sure people are aware of the location of their code. I was good about this up front, but I need to make it a part of the pre-requisities and slides. This is as simple as making sure you are in your Documents folder before cloning a repo.
  • Branching can get quite complex. I’m also not sure how much of a challenge this was for people to understand and perhaps because of this how exactly to address this complexity except by going through exercises (I even drew a few diagrams on a whiteboard). Ideally I’d have more information / follow up references and maybe another diagram or two. I don’t think I explained Remote vs local branches as well as I could have.
  • Not really an improvement but a general note of thanks: A few of the organizers from DAQAA had experience with git + GitHub and so I was happy to have had a few helpers roaming around. An unexpected but happy side effect was participants helping one another out which made my job that much easier.

The goal here was getting closer to the development workflow (like git + GitHub does) to help reduce the knowledge gap for testers, business analyst, other developers, etc. and help make them more comfortable with the technical aspects of software development. (There were a couple of attendees who were learning to program in Java and the course they were taking assumed they knew how to use Git + Github which they did not. This class helped them become more proficient.) Overall I think we achieved that goal.

More:

A GitHub Workshop & why Version Control is a Technical Skill

On April 11th, 2018 I’m giving a workshop in Des Moines, IA to the Des Moines Area Quality Assurance Association (DAQAA) called Contributing to GitHub Is for You (join us!) on learning to use git with GitHub. The workshop is based on a presentation I did with Matt Heusser called Contributing to GitHub is for Everyone from the OnlineTestConf last year.

One important technical skill of increasing importance is using version control (also referred to as source control) systems. For those un-familiar version control systems give the user the ability to track changes to code, text, html, images and pretty much any other file you want. This in turn gives the user the ability to accept, reject or restore changes to individual files on a granular level. Whether you want to look at code, documentation, file bugs, backup important files, or create a website you are increasingly likely to do this in version control.

At Laurel & Wolf people outside of our engineering team like Product, UX, legal have GitHub accounts just so they can have insight into what’s going on, what might be shipped, see what legal disclaimers we are using, etc. This is in addition to JIRA which is more broadly used across groups in the org.

1*iHPPa72N11sBI_JSDEGxEA

According to the latest Stack Overflow developer survey git is the most popular source control system in use today. It’s probably due in no small part to GitHub’s increasing popularity. (Or is that a chicken and egg problem?) Github is increasingly the place to host your public and private files and has given rise to the GitHub workflow aka create a branch, add commits, open a pull request, review code, and deploy. It’s a workflow that many companies and projects now use daily.

This means if you can use git to push changes to GitHub you can become technically fluent in basic software delivery. If it sounds interesting come join us or watch the shorter presentation mentioned above!

How to set up Apple Pay on Mac (non TouchID)

According to Bailey, Apple Pay availability was limited to about 3 percent of stores in the U.S. when it launched in 2014, but is now accepted in 50 percent of stores. – Courtesy of AppleInsider.

At Laurel & Wolf we use Stripe to handle monies we collect from our customers. It’s a great service that takes care of many things including allowing us to accept Apple Pay. Although accepting Apple Pay means we need to test it, which I’ve been doing mostly from my iPhone. However I’ve been wanting to test it on my laptop and that means enabling it on macOS for Safari.

These instructions are for non TouchID macs. If you have a Mac with TouchID you can go to Apple’s website for easy set up.

Apple Pay Logo

Steps to enabling Apple Pay on non TouchID Macs

  1. On your iPhone confirm Apple Pay is active. If not set it up here.
  2. On your Mac sign-in to iCloud using your Apple ID. Without this you won’t have access to your Wallet’s cards.
  3. Allow websites to check if Apple Pay is set up on Safari.
    1. In Safari go to the menu called Safari > Preferences > Privacy > look for the Apple Pay section. 
    2. Note: It can take about 5-10 minutes to propagate after you sign-in to iCloud.
  4. Done! Any website or app with Apple Pay enabled for checkouts will render an Apple Pay button as a checkout option. To test this you can go to Stripe’s Apple Pay website and test it out. (Don’t worry purchases happen in their test environment.)
  5. There are a few caveats:
    1. This only works in Safari
    2. You can only complete the purchase if your iPhone is connected to your Mac. Apple Pay requires TouchID to authorize the purchase.

Oh and if this article worked for you please consider sharing it or buying me coffee!

Review of The Selenium GuideBook: Ruby Edition

tl;dr If you’ve ever wanted to learn Selenium but didn’t know where to start, The Selenium GuideBook is the place (doesn’t matter which edition you use, it’ll be good).

 

Learning Selenium

The challenge of trying to learn Selenium WebDriver (simply referred to as Selenium) or any relatively new technology is that there is either too much disparate information (on the web), the material is out of date (many of the books I found) or of poor quality. With Selenium too many of the tutorials available to beginners had you using the Selenium IDE, which is a really poor option for writing maintainable and reusable test code. (See my previous post’s video for more on this.) I’ve walked out on conference workshops that sought to teach people to use the Selenium IDE to start their automation efforts. It wasn’t for me. I was going to do it right, I just had to figure out what that meant and where to start.

From the start I knew I wanted to learn about GUI test automation and more specifically Selenium WebDriver. I had tried WATIR (Web Application Testing in Ruby) and a few other tools but Selenium was open source and quickly becoming the go-to standard for automating human interaction with web browsers. It was and is the only choice.

Naturally I went searching the web for some tutorial or examples when I stumbled across several tutorials including Sauce Lab’s boot camp written by someone named Dave Haeffner. After struggling through the Bootcamp series (and finding some bugs in the process) I found Dave also produced a tip series called Elemental Selenium. I signed up for the Ruby weekly newsletter tips and went through many of the tips. Satisfied that Dave was worth learning from (good quality, relevant code examples) I decided it was time to try his book The Selenium GuideBook. I knew going into it, I was going to be the person maintaining the test suite and since I was more or less comfortable with Ruby I was happy The Selenium GuideBook came in that language!

Book Options

There are a few packages (book, code examples, videos, etc. ) for the language of your choice. As I said above I was more or less comfortable with Ruby so I ended up getting the “Ruby Edition, Just The Book” package. If I was doing this over today I probably would have done the “Cheat Sheets + Book” package and for JavaScript instead of Ruby.

The package itself contains a lot of great information and a number of materials:

  • The Selenium GuideBook; the Ruby edition is roughly 100 pages
  • Ruby code examples broken out by chapter
  • Elemental Selenium Tips eBook in Ruby
  • The Automated Visual Testing Guidebook

The first time I went through the book and code examples, it seemed redundant having different code for each chapter and example. It was only after I had gone through the chapters and examples for a second time, trying to apply and understand the differences that I began to understand the relative importance of seeing the code change chapter by chapter, example by example. The code examples all target an open source application Dave created called The Internet. It seems simple enough but many of the books and materials I went through either tried using Google or some badly written / hurried example.

The Book

Despite being less than 100 pages the Selenium GuideBook covers:

  • Defining a test strategy for your automation goals
  • Programming basics
    • Using page objects and base page objects to keep code clean
  • Locator strategies and tools
    • Relying on good locators seemed like a smart way to design tests. I wanted to avoid any record and playback tools and the poor locator strategies often employed.
  • How to write good acceptance tests
  • Writing Re-usable test code
  • Running on browsers locally and then in the cloud
  • Running tests in parallel
  • Taking your newly built tests, tagging them and getting them loaded into a CI server.

The whole package literally. In the preface Dave says the book is not full and comprehensive, it’s more of a distilled and actionable guide. The book really shows you how to start thinking about and building a test suite. It’s up to the user to take what they learned here and apply it to their application. That’s the fun part of the Elemental Selenium tips as well.

Applying the Book and Examples

After I had gone through all of the chapters and examples once, I went back through the relevant to me chapters and examples doing the following:

  1. Start with some very simple login tests.
    1. The book starts out this way as well. Writing tests that are not DRY or re-usable. but eventually get that way.
  2. Continuing through the code examples, getting a little more complicated and applying it to my own application.
    1. As I built out tests and start to see commonly repeated patterns, abstract out repeated code into relevant page objects. Eventually getting to a base page object.

In hindsight the hardest part of applying the book was trying to understand and apply a locator strategy within our application. While The Internet target application is great, it’s also a bit simplistic. Good for teaching, hard for bridging the sample application to the target application. Our single page .NET application was far more complicated and it took several attempts before I understood how my own strategy would work.

The transfer problem is always difficult. I mean how do you take what you learned and apply it to a slightly more sophisticated problem? It’s a difficult problem, not really a criticism of this book. It’s worth noting that whenever I had questions about what was written in the book, found a bug or two, or got stuck I could email Dave and within a week or so get a helpful response back.

Mission Accomplished!

With the lessons in this book and the Elemental Selenium Tips I was, through some focused time and lots of iterations, able to get a fairly good set of automated acceptance tests running against our application.

In other words, I highly recommend you buy the book. It’s slightly more expensive than similar books about Selenium but it’s far more effective. You are also directly supporting the author and with free updates and email tech support I think its well worth the cost.

Don’t believe me? Watch this video of Dave giving an overview of the GuideBook:

Additional References:

  1. http://www.tjmaher.com/2015/06/spotlight-on-dave-haeffner.html
  2. http://knowledge-anxiety.blogspot.com/2015/06/the-selenium-guidebook-and-thoughts-on.html
  3. http://davehaeffner.com/works/
  4. https://seleniumguidebook.com/

Selecting a few Platform Configuration Tests

I’ve been developing a GUI acceptance test suite to increase the speed of specific types of feedback about our software releases. In addition to my local environment I’ve been using Sauce Labs to extend our platform coverage (mostly by browsers and operating) and to speed up our tests by running more tests in parallel.

This pretty similar to what I consider traditional configuration testing – making sure your software can run in various configurations with variables such as RAM, OS, Video card, etc. Except on the web the variables are a little different and I am more concerned with browser differences than say operating system differences. Nevertheless, with over 700 browsers and OS platforms at Sauce Labs I still need to decide what configurations I start with and what configurations I add over time in the hope of finding failures.

Data

I figured the best place to start was with our current users and since the only “hard” data we had comes from Google Analytics, I pulled up the details of two variables (OS and Browser). Using TestingConference.org as a replacement for my company’s data, our most commonly used platform configurations include:

  • Browsers:
    • Chrome 47 & 48,
    • Firefox 43 & 44,
    • Safari 8 & 9, and
    • Internet Explorer 10 & 11
  • Operating Systems:
    • Windows 7,
    • Windows 8.1,
    • Windows 10,
    • Mac OSX 10.11 and
    • Mac OSX 10.10

Excluding a few constraints like IE only runs on Windows and Safari only runs on Mac, testing browsers and operating systems in combination could potentially yield up to 40 different configurations (8 browsers x 5 operating systems). Also, we designed our application to be responsive and at some point we probably want to test for a few browser resolutions. If we’ve got 40 initial configurations and 3 different browser resolutions that could potentially yield up to 64,000 different configurations. Realistically even for an automated suite and with one to two functional tests, that’s too many tests to run.

Reduce the number of tests

We can start with adding in those constraints I mentioned above and then focuses on just the variables we want to ensure we have coverage of. To get a better picture of the number of possible configuration tests I used ACTS (Automated Combinatorial Testing for Software) tool and based my constraints on what Sauce Labs has available today for configurations. After I added OS and Browsers it looked something like this:

ACTS Window

If I wanted every browser and operating system in combination to be covered (all-pairs) then according to ACTS there aren’t 40 different configurations, just 24 configuration options. That’s more manageable but still too many to start with. If I focus on my original concerns of covering just the browsers, I get back to a more manageable set of configuration options:

Test OS Browser
1 Windows8.1 Chrome47
2 Windows10 Chrome48
3 OSX10.11 Firefox43
4 Windows7 Firefox44
5 OSX10.10 Safari8
6 OSX10.11 Safari9
7 Windows10 IE10
8 Windows7 IE11

8 configuration options are way more manageable and less time consuming than 24 options, considering each one of those configurations will each run a few dozen functional tests as well.

A Good Start

Selecting the configurations is probably the easiest part of configuration testing (and the only thing I’ve shown) but I’ve found its worth thinking through. (The harder part is designing the automated acceptance tests to produce useful failures.) Using ACTS at this point may seem like overkill when we could have just selected the browsers from the beginning but it didn’t take much time and should make it easier in the future when we want to add more variables or change the values of our existing variables.

When to use a Gemfile

I’ve been building a GUI acceptance test automation suite locally in Ruby using the RSpec framework.  When it was time to get the tests running remotely on Sauce Labs, I ran into the following error:

RSpec::Core::ExampleGroup::WrongScopeError: `example` is not available from within an example (e.g. an `it` block) or from constructs that run in the scope of an example (e.g. `before`, `let`, etc). It is only available on an example group (e.g. a `describe` or `context` block).
occurred at /usr/local/rvm/gems/ruby-2.1.2/gems/rspec-core-3.2.2/lib/rspec/core/example_group.rb:642:in `method_missing'

It took a few minutes debugging before I spotted the error:

../gems/ruby-2.1.2/gems/rspec-core-3.2.2/lib/rspec/core/..

Source of problem: My remote tests were using a different version of RSpec than I was locally. Solution: Create a Gemfile to specify the version of using Rspec I’m using.

Since I didn’t realize I needed a Gemfile my question was, in general, when should someone use a Gemfile? According to the manual, a Gemfile

describes the gem dependencies required to execute associated Ruby code. Place the Gemfile in the root of the directory containing the associated code.

For example, in this context, I would place a Gemfile into any folder where I specifically want to call tests to run. In my case that meant a few specific locations:

  • At the root of the folder – where I run the whole suite of tests
  • In the /spec/ folder – where I typically run tests at an individual level

At a minimum I’d specify:

  • A Global Source
  • Each Gem I use locally that Sauce Labs will need to use

In the end it might look something like this:

Test, adapt, and re-test.