Good Tests vs Bad Tests and why you shouldn’t repeat them

For my second Racket I ranted a bit on this concept of Good vs Bad Tests and whether a good test (case) is a repeatable one.

Here’s the imperfect transcript:

Hi everyone, Chris Kenst here.

So my topic today is I want to talk about the difference between a good test and a bad test.

There’re two reasons that I bring this up today:

  • One I saw yet another article talking about how to create effective test cases and then going on to say they should be repeatable, which is of course, wrong your test should not be repeatable.
  • Two I have been asking this question as I hire for my third software tester here at Promenade group.

I am hiring somebody that’s going to be a senior tester. And I think one thing that a senior-level person should be able to do is differentiate between what makes a good test good and a bad test bad.

But so far in the, I want to say 15 or so, maybe 20 at this point candidates that I have asked this over email, very few have been actually able to describe what makes a good test and what makes a bad test. Most candidates seem to fall into this trap of what I saw in this article which is just bad advice where every test is a scenario regardless of what it is that you’re doing.

And this strikes me as very odd because it’s deeply violates this understanding of providing value.

So if you were to hire somebody to do a job and regardless of how the job changed over time, they were to do it the exact same regardless of changing circumstances, you would think that’s a bad candidate. Somebody wouldn’t want to hire.

It’s the same thing with test design. A good test versus a bad test is about value and it’s about focusing on what that test is going to do: its Mission. And so you can’t have the same approach to every kind of test because that means you’re not providing value. You’re not actually aiming to achieve your mission with the testing.

So if you see advice, or you read advice about good test versus bad test and it’s like you should make sure that all your tests are repeatable that’s just not the case.

For example, with boundary analysis and equivalence class partitioning, you don’t want repeatable test because they’re not built to be repeatable. Sure you could repeat them but now that test is no longer very useful and it’s probably not worth running again.

So it’s really challenging.

I will probably write an article about this too but if you’re ever thinking about how to write good test versus bad test, think about the value and think about the mission of the test itself. Then, try and think of all the different types of attributes that might make something valuable now, and something less valuable.

And then focus, on those things that deliver value, whether their power, whether the repeatability, whether they’re easier for coverage, for understanding that, because the better that we get as a community and better, we become better at testing will see that there are lots of different ways we can run tests.

And the more we understand that the more skilled and knowledgeable about our craft would become and this other kind of cool thing is that you can really set yourself apart from other people who may be interviewing.

If you can easily tell someone what a good test isn’t a bad test. If you are looking at someone’s test and going these are all scenario test you built the same thing just over and over again. How come you’re not varying anything?

All of a sudden you are in a much better position, out of all the other candidates because you can easily differentiate your work from their work and you can tie it to value.

So, that’s my rant today about good test versus bad test.

Have a great day, everyone.

Testing Community of Practice (CoP) Experience Report #1

I published an audio experience report on Racket.com about running my first Testing Community of Practice (CoP) at work. tl;dr it was a really good exercise that I intend to run regularly.

Here’s an imperfect transcript:

Hello everyone, Chris Kenst here, I wanted to talk about running my first community of practice.

So for a little context, I work for a company called Promenade group.

We have 4 verticals or businesses basically that we support. And so as a QA engineering manager, I have testers across two of those verticals.

So I have a tester across our vertical called BloomNation which helps florists sell flowers online, individual small businesses sell online.

And then we have a business of vertical called Swigg where I also have a tester.

So we have built out this team. Only two people hiring my third And so, as a part of a growing team, what I wanted to do was bring my testers together and just talk about the things that we’re doing. Because as we scale up, as we add a third tester in, perhaps a fourth one, each person kind of works in their own isolated business, tackling the challenges for that particular business.

And, so I really wanted to get this kind of cross collaboration system going.

So I thought a community of practice would be the right thing for it.

And frankly after I set up a calendar time, last Friday, I told my people about it, two weeks in advance, I asked to present something.

So they would present something and I would present something. I set an agenda for it and kind of got their feedback and nobody really knew what to think about it because they have never really done one. I have never really done one.

So we kind of all agreed to it and then on Friday we had it, I set up two hours on the calendar, and we use that whole two hours for just three people.

And so this is kind of how it went:

  • The first thing we did was I had each of my testers present, what they were working on.
  • So what have they worked on recently?
  • What challenges have, they had what things have gone well and what things happened?

And even only having two testers that I, of course, have one-on-ones with what happened, is having them discuss the things that they work about and the specific problems unique to the businesses that they support, it was actually really eye-opening even for me,

One of my testers works on our BloomNation business and that’s a pretty stable mature business. So she works on all these different things, spanning all these different features that don’t exist on our Swigg vertical. And so part of her discussion about what she’s working on is actually, educating the other tester about, hey, these are the unique things that we do over here.

And it’s like, no we don’t and then it was the vice versa.

It was the tester I have on our Swig vertical showing going like, oh listen, you know, we’re changing our checkout flow, we’re cleaning this up.

We’re integrating with this new partner that can help us do deliveries and so just having the two distinct teams talk about what they worked on it actually highlighted a whole lot of things that I have taken for granted because I have worked on Both platforms and so just kind of allowed a bit more common sharing.

It was also really impressive to see them, put together like a little presentation. Some with slides, some not with sides but it was really on point. Everyone was able to kind of talk to the things that like went.

Well, it was a little bit harder to kind of dig up dig deep and understand things that didn’t work well.

But I like the idea of at least getting I am thinking about hey what isn’t working well, because, you know, are these things that we need to fix within the verticals self, or are there, are there lessons that we can pick up and address just ourselves.

So like one of the things that we can do ourselves that kind of came out of this was, you know, we just need more documentation around some of the things that we test that are across both platforms. So that make sense. like we can create some, some Asks, and write a bunch of documentation, like, that’s something I can do.

And, so I think my feedback might my thoughts after going through it was clearly, it has value to do to have a small community of practice.

It would probably only run it every few weeks. Sorry every few months. Once a month would be too frequent. So probably every other month is good, there are clearly lessons and ideas to be shared.

One of my testers, I noticed is doing much deeper work because they’re focused on say an EPIC at a time, or my other testers doing much more broad things. So, they’re very different in terms of, you know, even within the same company on different teams, just very different kinds of work.

You know, one of my testers is more likely to look into New Relic logs for the things that she’s doing. And my other testers more likely to look into the third party logs. Because that makes sense, it’s just all kinds of very interesting things that I pull.

I came away with the other thing I noticed is that, of course I like to use zoom because it has better pictures better? It’s just better visual like you can see people better but it just doesn’t work when you only have 43 minutes and you have a two-hour meeting. So that was a take away from me.

Yeah, and so overall I think things worked really well. I hope to be able to run this again.

Do something similar where we talk about the things that worked well and the things that don’t and I think just as we grow community of practice and getting each other talking to one another, about the things that were working on and struggling with I think well, We will definitely have more benefits.

So, I will update you when I do my second one. Thank you.

Better Tester Training Materials

Photo by Clark Tibbs on Unsplash

Last month the Association for Software Testing (AST) announced a new partnership with Altom, the owner of BBST®, that enables the AST to refresh our curriculum lineup with the new BBST® Community Track and help fund the future growth of the materials. This partnership and refresh are a huge milestone for the AST. 

For some perspective: Cem Kaner developed the BBST courses over a long period of time, starting with Hung Nguyen in the 90s. In the 2000s after Cem Kaner was recruited to Florida Institute of Technology he and Becky Fielder received grants from the National Science Foundation to adapt them into online courses. Some time later Cem began collaborating with the AST to teach and develop the courses further. Those courses became known as AST-BBST to differentiate the way they developed and were taught (by passionate volunteers). Eventually Cem formed his own company, developed BBST® further and sold it to Altom after he retired. 

BBST® classes are well known for their depth of core testing knowledge and focus on improving through peer review work. That’s great for students but frankly it’s a maintenance challenge. Long before I started with the AST the problem has been, how do we properly maintain and evolve the materials and classes?

(more…)

Hiring a Software Tester, an Analysis

In May of 2020, back when Promenade Group was still called BloomNation, I opened a job posting for a Software Test Engineer. This was to be the first of many test positions we eventually hire for. After going through the whole process of hiring a software tester, I thought it would be useful to analyze the applicant data with the idea of learning something about how I hire and about the applicants who applied.

About the data

Some of this data was collected through our recruiting system and some was manually entered in by me. I spent a good deal of time crunching through raw data in Excel, then coming up with new questions and going back to find more data. Some of the data wasn’t captured at all and so I made guesses / assumptions. Specifically I did this for the applicants location and gender. I don’t hire based on gender, but I was curious to see how this might have effected the final outcome. Despite having 142 submissions, I ended up pulling data on only 107 resumes.

(more…)

I’m Speaking at TestFlix

It’s true, I’m speaking at TestFlix on November 28th, 2020. You should sign up to join; it’s free to register!

I recorded and submitted my 7 minute talk on “Using Test Idea Catalogs for Better Testing”. The premise is:

Testers can develop a set of tests or test concepts for a specific object or risk and re-use them in similar projects or products. Catalogs come in many shapes and forms, they can be lists or more detailed. They can be public or private. They can be developed by individual testers or as teams within companies. But they all help you test better!

Slides

References

I plan to write more on the topic of Test Idea Catalogs in the near future but I mentioned a few in the presentation that I’d like to call out here:

The TestOpsy

Back in January I hosted James Bach and Michael Bolton for an AST webinar on the concept of a TestOpsy or a way to learn about the testing you do by dissecting it. Below you’ll find not only a description and the webinar video but a transcription for what I hope is easier reading.

By looking very carefully at what you actually do, identifying your own heuristics, and putting that process into descriptive, evocative words, you can discover surprising depths in each act of testing you perform. In a testopsy, you build your skills of observation, narration, and test framing. You may even discover a technique no one yet has written about. And if you do it with a colleague, it stimulates discussion on test design.

James Bach & Michael Bolton

What do we mean by a Testopsy

We are talking about an autopsy for testing. We are talking about taking a very close look at a session of testing and you can do a testopsy based on just a few minutes of testing.

(more…)

I’m running for the 2020-2022 AST Board of Directors

Elections just opened for the Association for Software Testing’s Board of Directors for which I’m a candidate. If you are a voting-eligible member of the AST I’d appreciate the consideration as I run for my 2nd term.

For those who are voting (or possibly just interested) I completed a list of candidate questions. For fun you can compare them to my answers from 2018.

I did want to highlight a few questions that I think are important:

(more…)

My First Term on the AST Board of Directors

The Association for Software Testing (AST), a non-profit professional organization dedicated to advancing the understanding and science of software testing, has announced a call for nominations for the Board of Directors for 2020-2022. This means my two-year term as a director is coming to an end. I feel fortunate and grateful to announce I’m running for a second term. The AST has helped a lot of people including me. For this and a few other reasons described below, it feels like the right moment to reflect on what it has been like to help run this global non-profit.

The Golden Ticket

I was elected in August of 2018 while attending the Conference for the AST (CAST) in Melbourne, Florida. An AST member since 2012, I started volunteering in 2013 after I became an AST-BBST Instructor. Coming up through BBST, I thought educational advocacy was one of the AST’s most important community services. You can’t advance the understanding of the craft until testers have a solid understanding of what already exists. I really wanted to improve our offering and felt the best way was to help set priorities at the board level. 

AST Board of Directors

Elections happen every year with roughly half of the 7 person board up for election each year. The election process starts with a call for nominations and then candidates introduce themselves via questions posted to the web. Finally voting takes place during the time of CAST (typically the first week of August) and on the final day of the conference a new board is announced.

As a member-elect you are elected to a position by the sitting board members based, in part, on your preferences. In 2018 during a discussion with existing board members it came up there was a need for someone to take on the Treasurer position. It wasn’t the role I initially wanted (VP of Education was my first choice) but I felt reasonably competent so I accepted.

As with any official board position it’s a starting point for your contributions. I really wanted to focus on education but my fellow AST-BBST instructor Simon Peter (with whom I taught countless classes) wanted the position as well. We quickly both decided it made sense for him to take VP of Education and I take Treasurer. Just like we had done in our teaching we decided it would be fun to collaborate on the many changes we wanted to see in AST-BBST. I had my official role, Simon had his and yet we worked together whenever we could to improve our educational program.

(more…)

Regression testing isn’t only about repetition

Often when I’m chatting with someone about their regression testing strategy there is an assumption regression is all about repeating the same tests. This is a bit problematic because it ignores an important aspect which testers tend to be good at: focusing on risk. A better way to think of regression testing is it can be applied in two different ways: Procedurally and Risk-Focused

Procedural Regression Tests

When I speak of procedural I mean a sequence of actions or steps followed in regular order.  As I said above this seems to be the primary way people think about regression testing: repetition of the same tests. This extends to the way we think about automating tests as well.

Procedural regression testing can be quite valuable (so far as any single technique can be). The most valuable procedural regression tests are unit tests when applied to our CI system and run regularly. In this way they become a predictable detector of change, which is often why we run regression tests in the first place. (Funny enough automated UI tests are some of the most common procedural regression tests but aren’t the best detectors of change). 

The big problem with procedural regression tests are that once an application has passed a test, there is a very low probability of it finding another bug. 

Risk-Focused Regression Tests

When I speak of risk-focused I mean testing for the same risks (ways the application might fail) but changing up the individual tests we run. We might create new tests, combine previous tests, alter underlying data or infrastructure to yield new and interesting results.

To increase the probability of finding new bugs we start testing for side effects of the change(s) rather than going for repetition. The most valuable risk focused regression tests are typically done by the individual testers (or developers) who know how to alter their behavior with each pass through the system.  

A Combined Approach

Thinking about regression testing in terms of procedural and risk focus allows us to see two complementary approaches that can yield value at different times in our projects. It also gives testers an escape from the burden that comes  with repetition while still allowing us to meet our goals.

On other platforms:

If you liked this article please consider sharing it or buying me a coffee: