A few months ago I was chatting with Evgeny Kim about some of the reservations I had while exploring a new codeless test automation tool. He was also exploring some codeless tool options and so he invited me onto his podcast to talk about it. We chatted about a wide range of things such as challenges faced by software test engineers, the role of codeless tools and hiring problems.
It was a fun podcast with some interesting topics so I had the video translated. Watch the video or read the transcript whichever you prefer. (Sorry in advance for any mistakes.)
EVGENY KIM: All right, guys. Thank you for joining for today’s podcast session. We have a guest, Chris Kenst. He’s a QA Engineering Manager at a company, Promenade Group. Also, he’s a president of Association for Software Testing. And we’re going to talk today about the future of software test engineers. Thank you, Chris. Hi.
For the last few months I’ve been using a no-code UI test automation service called Reflect.run to build out some UI tests (scenarios and such) with the goal of evaluating how well it works in in terms of feedback (and value) as part of our build process. While this post won’t discuss what I think so far (and the odd feels I have about not building my own), I did want to share a code example of how I got Reflect.run builds to run in our CI pipeline.
Our current CI tool is called CodeFresh. Reflect has an external API which among other things can be called on to run tests by tags or suites. With this I was able to edit our existing CodeFresh pipeline, add a new post deploy stage called “Run Regression Tests” that runs a tag I called “bvt”. (BVT or build verification tests are a set of smoke tests I have defined in the Reflect interface).
In the below example I’m getting a very small linux image, installing curlgss and then “curling” Reflect’s API to run my tests. (If you don’t install curl first, you can’t make the curl call.) Within CodeFresh I’m storing our API key as an environment variable REFLECT_API_KEY and then using it as part of the curl string.
That’s all there is to getting Reflect.run‘s tests to run as part of a CodeFresh build. Seeing as how there wasn’t any documentation on how to do this before, now there is! (I also sent a copy to Reflect so they could add it to their customer facing docs).
One of the nice things about macOS are the built-in tools. A recently improved upon built-in tool is called Screenshot which allows users to take a screenshot and record video. Taking screenshots has been around macOS forever, but taking video directly from OS shortcuts came along in macOS Mojave. The challenge is remembering the commands.
This tutorial is an attempt to capture the quick commands / shortcuts to make this process easy.
When you are developing and/or testing a web app on iOS, having the ability to inspect and understand what is happening in the browser is incredibly useful. For just this reason Mobile Safari can use the same developer tools that are built into Safari on macOS.
To start debugging you’ll need your iOS device, a mac and a lightning cable. Then start by making sure both devices are configured to allow this.
Launch Safari on your mac; open preferences and within the advanced tab click the checkbox for “Show Develop menu in menu bar”.
Confirm, you should now see a menu called Develop in the top bar
On your iOS device (iPhone or iPad) go to Settings > Safari > Advanced and slide the toggle on for Web Inspector.
Debug a site in Mobile Safari:
Connect your iOS device (iPhone or iPad) to your mac
On the device browse to your site in Mobile Safari
On your mac’s Safari, under the Develop menu you should see an option for your iOS device (either called iPhone or iPad).
In that sub-menu you should see the url of the website you want to debug. Click that url and it will open a new Safari window with the full Safari Dev Tools.
Start your investigation!
A few weeks back I was testing changes to review submissions (buy a product, leave a review) and on Chrome desktop everything worked fine (including Chrome’s mobile emulator). However after testing on a real device, Safari didn’t behave the same way. Luckily we were able to debug using this same method and fix those remaining issues before any customers saw problems.
It just goes to show real device testing matters!
Oh and if this article worked for you please consider sharing it or buying me coffee!
Inspired by Justin Rorhman’s post of a similar name with a slight twist focusing on tools that generally help accelerate my testing.
As a test and quality specialist embedded in an engineering team I have a lot of work to do on any given day. Our engineering team’s goal is to ship quality software on a regular basis to deliver value to our customers. Naturally I rely heavily on a number of tools to help me better understand what is going on with the software or code under test and/or to accelerate what I’m able to do in a given amount of time.
Like Justin said in his article “Testing tools don’t make me a good tester, but they do make me a better tester. These tools help amplify things I need to do repeatedly or give me access to the product I wouldn’t have otherwise. They also shape the way I think about testing problems, for better and worse.”
In no particular order these tools are:
GitHub. We use the GitHub workflow to deliver all of our code including application, library, and test automation. With each push to a particular branch our CI system builds the application and runs our unit and integration tests. With each Pull Request I can see all or just specific changes, code reviews and test run summaries. All this helps keep our build pipeline clean, makes it easier to identify potential issues quickly and helps me quickly build my test strategy.
Docker. Each separate application of our system has been dockerized (aka runs in a container). This makes it incredibly easy to set new people up (a few hours of setup instead of a day) and gives us some powerful logging for each application. It also reduces conflicts when switching between branches where libraries or decencies have been changed.
Heroku review apps. As I mentioned above with each push our application is compiled through our CI system. Once finished our CI system pushes that complied build to Heroku which automatically provisions a staging like environment accessible through its own url. (This feature is called a Heroku Review application). In about 5 minutes after a set of changes have been built our team now has a brand-new test environment ready for those hard to test local problems like testing on real mobile devices or a few special integrated services. Being able to build these new automatic quickly allows our developers to show off / get quicker input to the work they are doing.
Browser DeveloperTools. I primarily use Chrome and Safari but the developer tools in each browser are priceless. From Inspecting requests and looking at their data, to setting or removing experiment variables, debugging problems, looking for problems, profiling, responsive design, layout, checking DOM elements, etc browser DeveloperTools really are the first place to start. It would be so much harder to learn what is going on without them.
CircleCI. Great for keeping your deployment pipeline clean, running your build tests and ensuring those changes get where they are going each time. It doesn’t much matter what CI system you use it’s helping to accelerate something that would normally take away from developer or tester time.
EverNote. Surprising or not, I’m always taking notes about any number of things like quick terminal commands, operations for creating data, migrating databases in heroku, setting up configurations, test ideas, questions for follow up, etc. Anything I might forget goes here and I can get to it anywhere, any place on any device.
iTerm2. A command line replacement utility with Oh My Zsh installed that I’ve customized and continue to customize with shortcuts and aliases to make my repetitive tasks quicker. Also customized with visuals showing which repo I’m in, what branch I’m on and if there are uncommitted files, because I do enough on the command line.
Automated UI acceptance tests. Running our acceptance tests for each staging deploy (or set of deploys) allows us time to do more deep exploration on the application as a whole with a reduced focus on those areas covered by the existing tests.
Part of the desire for writing this was to record what my work looks like now and then be able to compare it later-on in the future. However, in the time since I started writing this article I changed jobs and now the future is here. Well, kind of. My new team’s development stack is a lot more varied, which means so are the tools I’m using. I think it’s still best to wait for half a year or so before I write a follow up post on the tools accelerate that work. In the meantime, what tools do you use that accelerate your testing?
On April 11, 2018 I gave a workshop called Contributing to GitHub is For You is For You (abbreviated as C2GI4U) in Des Moines, IA to the Des Monies Area Quality Assurance Association (abbrevitated as DAQAA). Roughly 36 people signed up for the DAQAA event meetup and of those about 22 showed up. Scheduled for 3 hours (including a break in the middle for food) I set out to cover a number of topics around using Git + GitHub including:
Submitting changes in Pull Requests
Merging, reverting, cloning, forking, and other common commands, etc.
Using git locally from the command line and GitHub remotely through the web
In a little less than 3 hours we were able to run through three quarters of the exercises, covered all of the basic git commands and many of the intricacies of using the Git + GitHub workflow. Everyone walked away knowing how to grab and contribute to a project in Github using Git in the command line. A few people were overwhelmed having to pick up terminal commands in addition to Git commands, but most were able to get through the exercises and commit jointly to the C2GI4U repository. By the end a few felt so empowered they were going to apply what they learned the next day at work. (This point made me feel quite well.)
Workshops (generally) are fun because we get to do hands-on technical things. They also bring up interesting challenges, some of which are expected and others not so much. For example I expected some people might have to learn to navigate the command line (or re-familiarize themselves). Although we split time between GitHub’s nice UI you can’t get away from the command line but with practice during the workshop many people became more comfortable. However there were a few things that I didn’t expect and hope to improve upon the next time:
Provide a list of git commands we will use alongside the exercises. I had a list of all the git commands we were using as well as a list of exercises BUT they were in separate places.
I included a few cheat sheets with a bunch of different git commands but these were too broad to teach from, especially for people brand new to git.
Include information about storing GitHub up credentials. Although this wasn’t a big issue (you get a prompt to login) I always forget this step since my credentials are saved in my terminal. Everyone ran into this problem.
Make sure people are aware of the location of their code. I was good about this up front, but I need to make it a part of the pre-requisities and slides. This is as simple as making sure you are in your Documents folder before cloning a repo.
Branching can get quite complex. I’m also not sure how much of a challenge this was for people to understand and perhaps because of this how exactly to address this complexity except by going through exercises (I even drew a few diagrams on a whiteboard). Ideally I’d have more information / follow up references and maybe another diagram or two. I don’t think I explained Remote vs local branches as well as I could have.
Not really an improvement but a general note of thanks: A few of the organizers from DAQAA had experience with git + GitHub and so I was happy to have had a few helpers roaming around. An unexpected but happy side effect was participants helping one another out which made my job that much easier.
The goal here was getting closer to the development workflow (like git + GitHub does) to help reduce the knowledge gap for testers, business analyst, other developers, etc. and help make them more comfortable with the technical aspects of software development. (There were a couple of attendees who were learning to program in Java and the course they were taking assumed they knew how to use Git + Github which they did not. This class helped them become more proficient.) Overall I think we achieved that goal.
One important technical skill of increasing importance is using version control (also referred to as source control) systems. For those un-familiar version control systems give the user the ability to track changes to code, text, html, images and pretty much any other file you want. This in turn gives the user the ability to accept, reject or restore changes to individual files on a granular level. Whether you want to look at code, documentation, file bugs, backup important files, or create a website you are increasingly likely to do this in version control.
At Laurel & Wolf people outside of our engineering team like Product, UX, legal have GitHub accounts just so they can have insight into what’s going on, what might be shipped, see what legal disclaimers we are using, etc. This is in addition to JIRA which is more broadly used across groups in the org.
According to the latest Stack Overflow developer surveygit is the most popular source control system in use today. It’s probably due in no small part to GitHub’s increasing popularity. (Or is that a chicken and egg problem?) Github is increasingly the place to host your public and private files and has given rise to the GitHub workflow aka create a branch, add commits, open a pull request, review code, and deploy. It’s a workflow that many companies and projects now use daily.
This means if you can use git to push changes to GitHub you can become technically fluent in basic software delivery. If it sounds interesting come join us or watch the shorter presentation mentioned above!
Many eCommerce companies use Stripe to collect monies from customers. It’s a great service that takes care of many things (good test documentation) including allowing businesses to accept Apple Pay. Accepting Apple Pay means we need to test it, which I’ve been doing mostly from my iPhone. However I’ve been wanting to test it on my laptop and that means enabling it on macOS for Safari.
On your iPhone confirm Apple Pay is active. If not set it up here.
On your Mac sign-in to iCloud using your Apple ID. Without this you won’t have access to your Wallet’s cards.
Allow websites to check if Apple Pay is set up on Safari.
In Safari go to the menu called Safari > Preferences > Privacy > look for the Apple Pay section. (See the image below)
Note: It can take about 5-10 minutes to propagate after you sign-in to iCloud.
Note 2: You might need to connect your iPhone to your Mac for this option to appear.
Done! Any website or app with Apple Pay enabled for checkouts will render an Apple Pay button as a checkout option. To test this you can go to Stripe’s Apple Pay website and test it out. (Don’t worry purchases happen in their test environment.)
There are a few caveats:
This only works in Safari
You can only complete the purchase if your iPhone is connected to your Mac. Apple Pay requires TouchID to authorize the purchase.
tl;dr If you’ve ever wanted to learn Selenium but didn’t know where to start, The Selenium GuideBook is the place (doesn’t matter which edition you use, it’ll be good).
The challenge of trying to learn Selenium WebDriver (simply referred to as Selenium) or any relatively new technology is that there is either too much disparate information (on the web), the material is out of date (many of the books I found) or of poor quality. With Selenium too many of the tutorials available to beginners had you using the Selenium IDE, which is a really poor option for writing maintainable and reusable test code. (See my previous post’s video for more on this.) I’ve walked out on conference workshops that sought to teach people to use the Selenium IDE to start their automation efforts. It wasn’t for me. I was going to do it right, I just had to figure out what that meant and where to start.
From the start I knew I wanted to learn about GUI test automation and more specifically Selenium WebDriver. I had tried WATIR (Web Application Testing in Ruby) and a few other tools but Selenium was open source and quickly becoming the go-to standard for automating human interaction with web browsers. It was and is the only choice.
Naturally I went searching the web for some tutorial or examples when I stumbled across several tutorials including Sauce Lab’s boot camp written by someone named Dave Haeffner. After struggling through the Bootcamp series (and finding some bugs in the process) I found Dave also produced a tip series called Elemental Selenium. I signed up for the Ruby weekly newsletter tips and went through many of the tips. Satisfied that Dave was worth learning from (good quality, relevant code examples) I decided it was time to try his book The Selenium GuideBook. I knew going into it, I was going to be the person maintaining the test suite and since I was more or less comfortable with Ruby I was happy The Selenium GuideBook came in that language!
The package itself contains a lot of great information and a number of materials:
The Selenium GuideBook; the Ruby edition is roughly 100 pages
Ruby code examples broken out by chapter
Elemental Selenium Tips eBook in Ruby
The Automated Visual Testing Guidebook
The first time I went through the book and code examples, it seemed redundant having different code for each chapter and example. It was only after I had gone through the chapters and examples for a second time, trying to apply and understand the differences that I began to understand the relative importance of seeing the code change chapter by chapter, example by example. The code examples all target an open source application Dave created called The Internet. It seems simple enough but many of the books and materials I went through either tried using Google or some badly written / hurried example.
Despite being less than 100 pages the Selenium GuideBook covers:
Defining a test strategy for your automation goals
Using page objects and base page objects to keep code clean
Locator strategies and tools
Relying on good locators seemed like a smart way to design tests. I wanted to avoid any record and playback tools and the poor locator strategies often employed.
How to write good acceptance tests
Writing Re-usable test code
Running on browsers locally and then in the cloud
Running tests in parallel
Taking your newly built tests, tagging them and getting them loaded into a CI server.
The whole package literally. In the preface Dave says the book is not full and comprehensive, it’s more of a distilled and actionable guide. The book really shows you how to start thinking about and building a test suite. It’s up to the user to take what they learned here and apply it to their application. That’s the fun part of the Elemental Selenium tips as well.
Applying the Book and Examples
After I had gone through all of the chapters and examples once, I went back through the relevant to me chapters and examples doing the following:
Start with some very simple login tests.
The book starts out this way as well. Writing tests that are not DRY or re-usable. but eventually get that way.
Continuing through the code examples, getting a little more complicated and applying it to my own application.
As I built out tests and start to see commonly repeated patterns, abstract out repeated code into relevant page objects. Eventually getting to a base page object.
In hindsight the hardest part of applying the book was trying to understand and apply a locator strategy within our application. While The Internet target application is great, it’s also a bit simplistic. Good for teaching, hard for bridging the sample application to the target application. Our single page .NET application was far more complicated and it took several attempts before I understood how my own strategy would work.
The transfer problem is always difficult. I mean how do you take what you learned and apply it to a slightly more sophisticated problem? It’s a difficult problem, not really a criticism of this book. It’s worth noting that whenever I had questions about what was written in the book, found a bug or two, or got stuck I could email Dave and within a week or so get a helpful response back.
With the lessons in this book and the Elemental Selenium Tips I was, through some focused time and lots of iterations, able to get a fairly good set of automated acceptance tests running against our application.
In other words, I highly recommend you buy the book. It’s slightly more expensive than similar books about Selenium but it’s far more effective. You are also directly supporting the author and with free updates and email tech support I think its well worth the cost.
Don’t believe me? Watch this video of Dave giving an overview of the GuideBook: