Regression testing isn’t only about repetition

Often when I’m chatting with someone about their regression testing strategy there is an assumption regression is all about repeating the same tests. This is a bit problematic because it ignores an important aspect which testers tend to be good at: focusing on risk. A better way to think of regression testing is it can be applied in two different ways: Procedurally and Risk-Focused

Procedural Regression Tests

When I speak of procedural I mean a sequence of actions or steps followed in regular order.  As I said above this seems to be the primary way people think about regression testing: repetition of the same tests. This extends to the way we think about automating tests as well.

Procedural regression testing can be quite valuable (so far as any single technique can be). The most valuable procedural regression tests are unit tests when applied to our CI system and run regularly. In this way they become a predictable detector of change, which is often why we run regression tests in the first place. (Funny enough automated UI tests are some of the most common procedural regression tests but aren’t the best detectors of change). 

The big problem with procedural regression tests are that once an application has passed a test, there is a very low probability of it finding another bug. 

Risk-Focused Regression Tests

When I speak of risk-focused I mean testing for the same risks (ways the application might fail) but changing up the individual tests we run. We might create new tests, combine previous tests, alter underlying data or infrastructure to yield new and interesting results.

To increase the probability of finding new bugs we start testing for side effects of the change(s) rather than going for repetition. The most valuable risk focused regression tests are typically done by the individual testers (or developers) who know how to alter their behavior with each pass through the system.  

A Combined Approach

Thinking about regression testing in terms of procedural and risk focus allows us to see two complementary approaches that can yield value at different times in our projects. It also gives testers an escape from the burden that comes  with repetition while still allowing us to meet our goals.

On other platforms:

If you liked this article please consider sharing it or buying me a coffee:

I like working from home… 🏠

I like working from home… but it feels different now. 

Clarification: I like working from home, when it’s my choice.

My ideal work/life choice would involve: splitting the week into 2 or 3 days of in-office and from-home work. The extrovert in me would get to talk to my coworkers, pair up and communicate freely. The introvert in me would get to focus with limited distractions, kill my commute time and get to see my family a little more.

Back in the day I worked fully remote for over 3 years straight but this time I no longer have the same choices to keep me balanced. My old office became the kid’s bedroom and although I love my standing desk it’s also in the dining room. Staying inside for most of the workweek used to be fine when I could go out to eat for lunch or dinner or spend time with friends. Those options are much reduced today.

However even as we retreat from physical interaction I’m reminded there’s still the possibility to learn and connect with others. This morning I was Skyping with a friend in the Netherlands who had just gotten off of a random Zoom call with 30+ Strangers for an happy hour drink. Someone posted a link, others joined and enjoyed an after work drink together!

And that’s the point. There are still choices to be made despite our shelter-in-place orders. I take long walks with the family after work and on occasion I fly my drone to explore my surroundings. This weeks goal is to start taking walking breaks at regular interviews where I’ll listen to my podcasts and perhaps say hi to people doing the same. Who knows, maybe I’ll host or join social hangouts for fun?

Five for Friday – March 13, 2020

Welcome to Friday, here are five points worth exploring:

February 2020 Updates

This month has been busier than the last few and I’m feeling the need to chill. Part of this busyness is having just wrapped up teaching an AST-BBST Foundations class, which is about equally as intense for the instructors as they are for the students. The other part is at work, we are through our biggest week of the year aka Valentines Day (did you order your flowers?). Oh and my 49ers lost the Super Bowl. It’s fine, everything is fine now. At least I’ve settled in with my team being second best in the league this year. 

Come to think of it there are a few more things happened that are worth mentioning:

  • On March 20, 2020 I’ll be speaking at a new free online conference by Kobiton called Odyssey on the topic of How I Became A Test Automation Engineer. This will be an updated (read better!) version of the talk I gave at the OTC. The conference takes place March 19-20, 2020 and you can sign up for free!
  • I’m now the proud owner and maintainer of the Context-Driven-Testing website. This is the official CDT website created by Cem Kaner to share the CDT principles. Along with the principles comes a blog that I hope to add to over time. This is a huge thing for me as someone who takes a CDT approach to testing, to have this piece of history.
  • In some ways I consider myself a bit of a historian of the testing community, especially around the areas of the community where “I came up”, e.g. around CDT. If you saw last month’s update you’ll know I also manage the LAWST website, which although is about Peer Workshops, definitely comes from the CDT community. To me these are some prized possessions because they contain a lot of history, a lot of writings about the growth and challenges of software testing community in general.
  • It’s crazy to me that most of TestingConferences.org’s Sponsorships for the year 2020 are gone! I’m still working on a way to get Sponsorships set up here, but that’s on the long list of items in my backlog. Maybe next month I’ll get the time to do this? 

2019 in Review

My yearly tradition has been to summarize the most popular and important (to me) articles I’ve written over the past year along with some reflections and other forward-looking (and likely wrong) statements mixed in.

You can find previous years in review here: 2018 | 2017 | 2016 | 2015

Santa Monica Pier, near my office

What happened in 2019?

I joined BloomNation as a Test Automation Engineer in 2018 and in 2019 was promoted to senior role. The promotion was in part my testing contributions and hitting my goals but also the general impact I’ve had on the company outside of that role. During CAST I was telling people I had temporarily switched positions by taking on a Product Manager role until we could fill the vacancy. While temporary, I helped the company continue to deliver on a new core piece of the business and in turn it gave me some new experience and perspective. I plan to write about this experience soon. 

Speaking of CAST 2019, this was my first year helping to organize and run a conference of any kind. I plan to write about this experience as well but suffice it to say this was both fascinating and incredibly hard. 

I did some speaking in 2019, made my first and second podcast appearance and found what I hope is a sustainable model for supporting TestingConferences.org. I’ve started to become more serious about potential ways to support the things I do. In fact one theme of 2020 might be me figuring out how to balance all the things I want to do with all the things I’ve already committed to doing.  Just like every year!

The Five Most-Viewed Articles: 

  • How to debug problems on Mobile Safari – After showing a co-worker how they could debug mobile safari problems on their MacBook, I realized it wasn’t common knowledge. So I helped change that a bit but making it clear how to do it.
  • Participating in Code Reviews as a Tester – I’ve always liked the concept of helping testers push their technical understanding to reduce both risk and increase confidence. This post was based on a webinar of the same name and includes the slides and a link to watch the webinar.
  • How I Became an Automation Engineer – A talk based on my personal experiences of becoming an Automation Engineer and what my role looks like. This also became a blog post with references, slides and an embedded video.
  • Move Fast and Make Things Better – I much prefer the saying move fast and make things better over move fast and break things and this article points out why.
  • It’s Easier to Write about Tooling – Whenever I go to conferences there is a heavy emphasis on what our tooling is. Even in my own writing there can be a heavy emphasis on tools and I think that’s just because it’s easier than writing about the decisions we took and models we made prior to choosing it.

Over 185k page-views in 2019

The first article made it into my top 10 articles over all, which is great. Traffic to this site continues moving up and to the right over time. In 2019 alone I had more than 185k page-views. That’s double the views in one year! Wow!

A Few Other Articles

  • It was the Creative Web that collapsed – The title of this post is a line from Edward Snowden’s new book, Permanent Record. It’s a great book, you should read it. I reflected on how Surveillance Capitalism is part of the world of the web and how I hope to limit that surveillance on all my sites.
  • Getting Paid to Learn was a reminder of how important it is to have the company you work for support your professional development. Making promises is one thing but taking action to do it has a positive effect on the people and the company culture.
  • You Don’t Have to be an Expert to Teach is a friendly reminder that all we need to do to teach someone something is know slightly more than they do.

These are articles were written because I was feeling the flow and when inspiration hits you take it. They turned out well, I’m proud to write consistently about topics that intrigue me and I hope there’s intrigue on the other side too. 

The Future is already here

Predicting the future is fun and yet meaningless. But here are the things already on my radar for 2020. So much to consider and so little time:

  • CAST 2020 is already rolling with early bird ticket sales open.
  • I’m almost done teaching an AST-BBST Foundations course, my first in a few years and I’m constantly looking at the course with an eye for improvements.
  • Speaking of improvements, I’m still helping to redesign some aspects of the course AND of course trying to be a treasurer and handle all the financial things.
  • When it comes to writing I hope to continue the pace of 2x blog posts per month. It’s challenging but doable. More would be great but I want a constant pace to keep myself going.
  • I also hope to start publishing / writing more articles that touch on JavaScript, CypressIO and WebDriverIO. I’ve been primarily using JavaScript in the last year in a half and I’ve written almost no examples using it.
  • Despite record setting viewership in 2019, I’m hoping to get better at self branding and sharing which should see page-views increase YOY again. I’ve already updated the newsletter to be less often so I can focus on driving readership for certain articles. I also intend to cross-promote on other blogging and reading platforms to gain viewership. So much to do.
  • I essentially did 3 presentations / talks in 2019 and I intend to do the same or a few more this year (especially for the online conferences). Hopefully I’ll also get around to making a few more podcast appearances because they are fun to do and fun to share.

Cheers to the rest of 2020! What will you be doing?

Five for Friday – January 17, 2020

Welcome to Friday, here are five points worth exploring:

  • Our first AST webinar for 2020 about TestOpsies is now live on the AST YouTube channel, check it out:

January 2020 Updates

  • Tomorrow, Wednesday, January 8th at 11am, I’m hosting James Bach and Michael Bolton for a webinar on TestOpsies – Dissecting Your Testing. Learn More or Sign up today to attend / receive the recording!
  • Starting on January 12th I’ll be teaching the first AST-BBST Foundations course of 2020. The class is sold out which is good and a little stressful but should be fun. It’s also the first time in more than a year that I’ll be teaching. Learn more about AST-BBST classes
  • CAST 2020 is going strong. We extended the CFP until January 10, 2020 to squeeze in more amazing speakers and concepts as possible. Early Bird tickets are on sale!
  • Half of TestingConferences.org’s Sponsorships are booked which is crazy! I’m happy people are interested in supporting the project and frankly this support is going to help us spend more time / build better features. 
  • I’m now the proud owner and maintainer of the LAWST website. LAWST is the Los Altos Workshop on Software Testing, and it’s format is widely known / popular peer workshops model. 
  • I’m attending the Automation Guild 2020 online conference coming up the first week of February. Honestly I like online conferences. While it’s not a guarantee I’ll watch the videos (I don’t have dedicated time), I like the convenience, content and low time commitment.

I’ve been reading Troy Hunt’s weekly update series and while I love the concept and execution of such frequent updates, I don’t think I have useful enough information to capture so frequently. I might however start posting videos as an option for those who don’t want to take the time to read.

It was the Creative Web that collapsed

In his book Permanent Record, Edward Snowden suggests the evolution of the internet has gone from a community without border or limit where “anonymity-through-polyonym produced more truth than falsehood” to one that is unrecognizable today. Unrecognizable, in part, due to the loss of individual websites shuttered by the promise of convenience and replaced with eCommerce platforms that lend themselves to surveillance capitalism.

He goes on to say:

“Now, it was the creative web that collapsed, as countless beautiful, difficult, individualistic websites were shuttered. The promise of convenience led people to exchange their personal sites—which demanded constant and laborious upkeep—for a Facebook page and a Gmail account. The appearance of ownership was easy to mistake for the reality of it. Few of us understood it at the time, but none of the things that we’d go on to share would belong to us anymore. ”

Edward Snowden, Permanent Record

It may have been a conscious choice to shutter individual websites in favor of something simpler (like publishing on a writing platform or even Facebook) but it’s also been a conscious choice that I haven’t.

Here and elsewhere I’d rather be in charge of my own personal sites than use someone else’s platform despite the potential ease and built-in audience. I recongize in exchange for putting up with this pain I get my own little piece of the web where surveillance is minimal (the only tracking here is through Google Analytics) and I get the freedom to do what I want and the opportunity to learn from the ownership of it all. Permanent Record is just one resource on learning what it means to minimize surveillance here and elsewhere.

More than anything else over the years it’s been this opportunity to learn that keeps me running my part of the creative web. For many this might sound like a headache but domain management, DNS, caching, SSL, website performance, SEO, uptime, etc. etc. are all opportunities to better understand the various parts of the internet and to become a more knowledgable consumer.

It’s also not uncommon for these duties to fall to the engineering team of a small company or startup and the more familiar you are, the better you can contribute.

Running my own beautiful, difficult, individualistic website is something I grew into over time and what I recommend others do as well. I don’t mind the difficultly given the upsides of what you can learn and doing my small part in limiting surveillance capitalism.