Website Update? Don’t forget to Annotate!

Don’t forget to annotate changes!

Let’s say you published new layout changes to your website or you released new content or even a new landing page. How do you easily record such events so you can see what happens to your traffic? If you are using Google Analytics you can add annotations!

Simply put, annotations are short user notes (up to 160 characters) in the interface. They don’t affect the back-end data in any way. Think of them as sticky notes on your reports. After all the data has been processed and pushed into reports, you can attach notes and comments for specific dates.

They are a great way to spot correlations of data with external events.

AnalyticsMarket blog post

I use annotations to provide context to the days I make big changes. While I don’t do any marketing events or run A/B tests, I do occasionally make layout changes or update menus. Here’s what my recent updates look like:

A snapshot of my annotations (redacted for enhanced suspicion)

To create an annotation in Google Analytics:

  1. On your Audience Overview
  2. Click the small tab below the timeline
  3. Click ‘+ Create new annotation’
  4. Select the date for the annotation
  5. Enter your “sticky note”
  6. Select the visibility of the annotation (public vs private)
  7. Save

A sticky note showing what updates I made and when me make sense of data changes and trends in retrospect. Before I created a status page, I would use make a note of site outages so I could judge how reliable my web host was. (I should probably still do this). There’s far more I could do to be proactive but this works for now.

How to take a screenshot and record video in macOS

One of the nice things about macOS are the built-in tools. A recently improved upon built-in tool is called Screenshot which allows users to take a screenshot and record video. Taking screenshots has been around macOS forever, but taking video directly from OS shortcuts came along in macOS Mojave. The challenge is remembering the commands.

This tutorial is an attempt to capture the quick commands / shortcuts to make this process easy.

(more…)

Building an Awesome Home Office Space

My Desk
My current awesome home space

In the United States it seems like most of us will be working from home at least until Q1of 2021. I was thinking about this the other day: when will we see infection levels low enough that masks will no longer be required AND such a decision won’t lead to a rebound of the virus? And… When will a vaccine be available and widely used? In other words… it’s going to be at least another 6 months.This isn’t meant to be depressing, rather quite the opposite. Knowing the work situation will be the same for the next 6 months gives me the opportunity to change things up so the next 6 months are better than the previous.

Nearly 4 years ago I had my own home office. It was a full room and I could keep the door closed to avoid distractions. Now my desk is in the middle of the living room and I have to deal with such a busy place. 

If I truly want to make my work situation better for the next 6 months what changes should I make to build an awesome home office space? Additionally, what can I do that is convenient and cost effective? The answer I found is to use as much equipment as I have access to from work and home.

(more…)

I’m running for the 2020-2022 AST Board of Directors

Elections just opened for the Association for Software Testing’s Board of Directors for which I’m a candidate. If you are a voting-eligible member of the AST I’d appreciate the consideration as I run for my 2nd term.

For those who are voting (or possibly just interested) I completed a list of candidate questions. For fun you can compare them to my answers from 2018.

I did want to highlight a few questions that I think are important:

(more…)

My First Term on the AST Board of Directors

The Association for Software Testing (AST), a non-profit professional organization dedicated to advancing the understanding and science of software testing, has announced a call for nominations for the Board of Directors for 2020-2022. This means my two-year term as a director is coming to an end. I feel fortunate and grateful to announce I’m running for a second term. The AST has helped a lot of people including me. For this and a few other reasons described below, it feels like the right moment to reflect on what it has been like to help run this global non-profit.

The Golden Ticket

I was elected in August of 2018 while attending the Conference for the AST (CAST) in Melbourne, Florida. An AST member since 2012, I started volunteering in 2013 after I became an AST-BBST Instructor. Coming up through BBST, I thought educational advocacy was one of the AST’s most important community services. You can’t advance the understanding of the craft until testers have a solid understanding of what already exists. I really wanted to improve our offering and felt the best way was to help set priorities at the board level. 

AST Board of Directors

Elections happen every year with roughly half of the 7 person board up for election each year. The election process starts with a call for nominations and then candidates introduce themselves via questions posted to the web. Finally voting takes place during the time of CAST (typically the first week of August) and on the final day of the conference a new board is announced.

As a member-elect you are elected to a position by the sitting board members based, in part, on your preferences. In 2018 during a discussion with existing board members it came up there was a need for someone to take on the Treasurer position. It wasn’t the role I initially wanted (VP of Education was my first choice) but I felt reasonably competent so I accepted.

As with any official board position it’s a starting point for your contributions. I really wanted to focus on education but my fellow AST-BBST instructor Simon Peter (with whom I taught countless classes) wanted the position as well. We quickly both decided it made sense for him to take VP of Education and I take Treasurer. Just like we had done in our teaching we decided it would be fun to collaborate on the many changes we wanted to see in AST-BBST. I had my official role, Simon had his and yet we worked together whenever we could to improve our educational program.

(more…)

FAQs to How I Became An Automation Engineer (talk)

I’ve given the talk on How I Became an Automation Engineer a few times. Each time I’ve gotten good feedback and a lot of questions from engaged participants. With each repeated question, I try to update my talk to address those points for future audiences. However often I do this some repeated questions still arise. I’ve taken this challenge as both an excuse to write out my answers and try out a FAQ schema block!

Frequently Asked Questions

Here are the FAQs for my talk How I Became an Automation Engineer.

How do you decide the scripting or programming language to use in your testing? Do you typically use the same language the app is built in?

My recommendation is to use a programming language that your company supports. By this I mean a language that someone within your company uses and can provide feedback on your code through code reviews. This may or may not be the language the application is built in. Most companies use several languages. You might build your tools in JavaScript while the application is built in Java.

Ultimately your goal is to tackle the task at hand. Use the tools and languages that make sense.

Is (insert name of your current tool) sufficient enough to perform all types of automation testing?

No, probably not. Hopefully you work for an organization that wants to use the right tool for the right job.

Part of the process for learning about tools is we figure out what they are and aren’t good for. No one tool is good for all jobs. The same goes for the person using the tool. If you only know one tool you’ll try to use it for every problem. The more tools we learn, the more likely we will use the right tool for the right job. This also makes us more valuable in the market for new jobs.

What are your thoughts on using frameworks with record features to start learning automation?

It depends on your needs and your skills.

In the past, Record and Playback features were sold as Snake Oil. They’d fix all your problems. But they usually were much good beyond proof of concepts.

Today, more and more services are cropping up with codeless / record and playback capabilities. If these meet the needs of your project and/or company, then use them. One challenge for the individuals learning and using these these tools is they might not benefit your (market) value in the long term.

Sometimes the team or project doesn’t support automation either for budget or time concerns. How do you tackle that?

Test Automation is there to provide feedback, but it costs money and takes time to build and maintain. So if you are on a short term project then it might not be worth the time an effort to do certain kinds of test automation.

What kinds make sense? Well it depends on the project. It might be worth it to do unit tests but not much else. Or it might be worth it to build some one-off probabilistic automated tests to help you discover problems because it doesn’t have to be maintained. Remember the point of automation is to reduce our time or extend our capabilities but all that must be done within the context of your project.

Do you recommend learning to test and automate at the API level? Is Postman adequate for this?

Yes! In general I recommend automating tests at the unit level first, then at the service or API level and lastly at the UI level. Learning to explore APIs is a big part of this and eventually once you explore them, you can create automated tests.

Postman is a good tool for exploring APIs (there are probably others). It might also make sense for you to write some automated tests in Postman but that really depends on your goals and/or if there are better places to do it. This depends on your overall strategy.

According to your last slide, the future of the Automation Engineer role is uncertain. Do you think it’s better to move to the developer role or there is still hope that automation roles will be still exist?

We know that in high performing teams developers own the automated tests and testers are allowed to explore for risks. So there’s a high probability in companies looking to build high performing teams that Automation Engineering roles will disappear or not exist. There’s also a high probability of developer roles being needed in the future. We just don’t know when any of this will occur. If you are going to move, it makes sense to survey the landscape and see what the options are, see what your skill levels are and be practical.

What is the name of the book about you recommend / quote about high performing teams handling test automation?

Want to watch the latest recording of this talk?

Five for Friday – April 17, 2020

Welcome to Friday, here are five points worth sharing. A few of these are “work” related but not all. Here is to helping find balance between working from home, home time and personal time:

  • I’m finishing my second week of One Hundred Push ups. It’s hard but fun and takes less than 10 minutes per day. I’ve hit a few walls (because I’m out of shape) but the next workout (I do 3x a week) I usually break down the wall and continue on.
  • Arnold Schwarzenegger has been busy on social media helping people focus on what they can control. Then he shared his old no gym / home workout routine using just bodyweight. It’s kind of legit. Men’s Health has it. I’ll try it after I can do 100 push ups!
  • For those parents who are looking for ways to plan their children’s education while being remote we just signed up for ABCmouse.com for our 2.5 year old. Between this and the iOS app Endless Learning Academy I think it’s going well.
    • We’ve actively been trying to experiment with learning plans to keep the kid developing. He doesn’t go to preschool yet but then again it seems like no kids currently do.
  • Tea-Time with Testers magazine has a new issue after over 2 years on hiatus. Included was an article that struck home with me called How To Read A Difficult Book by Klára Jánová and James Bach. I also have tried reading An Introduction To General Systems thinking several times and have not gotten what I wanted from it. Perhaps I need to try what Klára did.
  • There are some good bits of advice in the a16z podcast: Moving to Remote Development (and work). One is about the kinds of bad behaviors that get emphasized when moving to remote work (especially for managers). Another is a discussion around communication, using video, how to be expressive and generally send the right signals.

And a friendly reminder that with Amazon delivering slower than normal for non-essential things, eBay sellers are shipping faster than ever!

Upgrading to WebDriverIO 5

A few weeks ago I finished upgrading our implementation of WebDriverIO from version 4 to version 5. The impetus for the upgrade was an announcement by the WebDriverIO twitter account of a new beta version 6 to be quickly followed by a finished version (it’s already here). One thing was clear: you have to be on v5 to go to v6 and each new subsequent version would only be supported for a year. Time to upgrade!

I’d been on version 4 since I originally deployed WebDriverIO in mid 2018. I knew version 5 was out but I had no immediate plans to upgrade given all of the warnings around breaking changes.

Preparation

This isn’t to say I wasn’t preparing myself. I created a JIRA ticket to outline what the work might look like. I was going through the TestAutomationU course on WebDriverIO which uses V5 and of course practicing in my own repo. With confirmation of the EOL of version 4 it was time to move on. 

I scoped out more of the work in JIRA (and also made a story to upgrade to v6). I bookmarked a few important articles including the WebDriverIO blog post announcing version 5 which highlights, among other things, specifically How to Upgrade. Then I made note of the methods that were either changing or being replaced for easier reference.

Upgrading to WebDriverIO 5

Finally it was time for the upgrade itself. The first question I needed to answer was logistical: how do I approach making the changes? Create a whole new repo, install the new packages and then move my tests over? Or just upgrade in place? Despite the daunting feeling I had, I figured it would be easier to upgrade in place and deal with the test failing as they came. This would introduce less changes than actually trying to move things over. Then it was time to follow the recommendations in the blog post:

  1. Remove node_module
  2. Remove all wdio-* packages from package.json
  3. Install the latest version of webdriverio: $ npm install [email protected]
    1. Note: if you did this today, you’d want to npm install [email protected] however I’d recommend going straight to v6 instead of going to v5 and then v6.
  4. Install the new wdio testrunner: $ npm install @wdio/cli --save-dev
  5. I have multiple configuration files and so I didn’t need to back them up. Rather I created a new one as part of the webdriverio configuration wizard. Then I eventually migrated / pruned the original ones until they were what I wanted.
  6. Rerun the configuration wizard: $ npx wdio config

All the Broken Things

Bam! WebDriverIO 5 deployed. Kind of. The easy part was done, next up was running each test one at a time, starting with the easiest / shortest tests. (I usually start with low hanging fruit so that I build momentum). When a test failed, I’d find out where and why (due to a rename or deprecation) and make a change. Rise and repeat for the whole test suite.

If this sounds simple, that’s because it was. All it took was time to remember where things were and why something was done a certain way and then make a change. 90% of the time this worked fine.

Other times it was more complicated. In at least one instance I ran into a breaking change, the functionality that was there before didn’t exist in the new version. I commented out 2 tests and moved on. (Speaking of which, I need to open a bug report about this).

Benefits

While it’s never fun to take something that’s working and break it just to upgrade, it does have some side benefits. Such as…. I fixed a few remaining bad ternary statements. Then cleaned up some unused libraries / plugins that I installed for some reason but can’t possibly remember anymore. Also refactored a bit of my code to make things simpler. When I was going through the TestAutomationU course I got to see how the author structured her tests. Now I have a new task to break apart my larger tests into smaller more defined tests with less assertions. 

All of this is to say, once you start making changes to improve one thing, it can snowball and lead to even more improvements. 

If you liked this article consider sharing it or buying me a coffee:

Regression testing isn’t only about repetition

Often when I’m chatting with someone about their regression testing strategy there is an assumption regression is all about repeating the same tests. This is a bit problematic because it ignores an important aspect which testers tend to be good at: focusing on risk. A better way to think of regression testing is it can be applied in two different ways: Procedurally and Risk-Focused

Procedural Regression Tests

When I speak of procedural I mean a sequence of actions or steps followed in regular order.  As I said above this seems to be the primary way people think about regression testing: repetition of the same tests. This extends to the way we think about automating tests as well.

Procedural regression testing can be quite valuable (so far as any single technique can be). The most valuable procedural regression tests are unit tests when applied to our CI system and run regularly. In this way they become a predictable detector of change, which is often why we run regression tests in the first place. (Funny enough automated UI tests are some of the most common procedural regression tests but aren’t the best detectors of change). 

The big problem with procedural regression tests are that once an application has passed a test, there is a very low probability of it finding another bug. 

Risk-Focused Regression Tests

When I speak of risk-focused I mean testing for the same risks (ways the application might fail) but changing up the individual tests we run. We might create new tests, combine previous tests, alter underlying data or infrastructure to yield new and interesting results.

To increase the probability of finding new bugs we start testing for side effects of the change(s) rather than going for repetition. The most valuable risk focused regression tests are typically done by the individual testers (or developers) who know how to alter their behavior with each pass through the system.  

A Combined Approach

Thinking about regression testing in terms of procedural and risk focus allows us to see two complementary approaches that can yield value at different times in our projects. It also gives testers an escape from the burden that comes  with repetition while still allowing us to meet our goals.

On other platforms:

If you liked this article please consider sharing it or buying me a coffee:

I like working from home… 🏠

I like working from home… but it feels different now. 

Clarification: I like working from home, when it’s my choice.

My ideal work/life choice would involve: splitting the week into 2 or 3 days of in-office and from-home work. The extrovert in me would get to talk to my coworkers, pair up and communicate freely. The introvert in me would get to focus with limited distractions, kill my commute time and get to see my family a little more.

Back in the day I worked fully remote for over 3 years straight but this time I no longer have the same choices to keep me balanced. My old office became the kid’s bedroom and although I love my standing desk it’s also in the dining room. Staying inside for most of the workweek used to be fine when I could go out to eat for lunch or dinner or spend time with friends. Those options are much reduced today.

However even as we retreat from physical interaction I’m reminded there’s still the possibility to learn and connect with others. This morning I was Skyping with a friend in the Netherlands who had just gotten off of a random Zoom call with 30+ Strangers for an happy hour drink. Someone posted a link, others joined and enjoyed an after work drink together!

And that’s the point. There are still choices to be made despite our shelter-in-place orders. I take long walks with the family after work and on occasion I fly my drone to explore my surroundings. This weeks goal is to start taking walking breaks at regular interviews where I’ll listen to my podcasts and perhaps say hi to people doing the same. Who knows, maybe I’ll host or join social hangouts for fun?