In May of 2020, back when Promenade Group was still called BloomNation, I opened a job posting for a Software Test Engineer. This was to be the first of many test positions we eventually hire for. After going through the whole process of hiring a software tester, I thought it would be useful to analyze the applicant data with the idea of learning something about how I hire and about the applicants who applied.
About the data
Some of this data was collected through our recruiting system and some was manually entered in by me. I spent a good deal of time crunching through raw data in Excel, then coming up with new questions and going back to find more data. Some of the data wasn’t captured at all and so I made guesses / assumptions. Specifically I did this for the applicants location and gender. I don’t hire based on gender, but I was curious to see how this might have effected the final outcome. Despite having 142 submissions, I ended up pulling data on only 107 resumes.
I recorded and submitted my 7 minute talk on “Using Test Idea Catalogs for Better Testing”. The premise is:
Testers can develop a set of tests or test concepts for a specific object or risk and re-use them in similar projects or products. Catalogs come in many shapes and forms, they can be lists or more detailed. They can be public or private. They can be developed by individual testers or as teams within companies. But they all help you test better!
Back in January I hosted James Bach and Michael Bolton for an AST webinar on the concept of a TestOpsy or a way to learn about the testing you do by dissecting it. Below you’ll find not only a description and the webinar video but a transcription for what I hope is easier reading.
By looking very carefully at what you actually do, identifying your own heuristics, and putting that process into descriptive, evocative words, you can discover surprising depths in each act of testing you perform. In a testopsy, you build your skills of observation, narration, and test framing. You may even discover a technique no one yet has written about. And if you do it with a colleague, it stimulates discussion on test design.
James Bach & Michael Bolton
What do we mean by a Testopsy
We are talking about an autopsy for testing. We are talking about taking a very close look at a session of testing and you can do a testopsy based on just a few minutes of testing.
Elections just opened for the Association for Software Testing’s Board of Directors for which I’m a candidate. If you are a voting-eligible member of the AST I’d appreciate the consideration as I run for my 2nd term.
The Association for Software Testing (AST), a non-profit professional organization dedicated to advancing the understanding and science of software testing, has announced a call for nominations for the Board of Directors for 2020-2022. This means my two-year term as a director is coming to an end. I feel fortunate and grateful to announce I’m running for a second term. The AST has helped a lot of people including me. For this and a few other reasons described below, it feels like the right moment to reflect on what it has been like to help run this global non-profit.
The Golden Ticket
I was elected in August of 2018 while attending the Conference for the AST (CAST) in Melbourne, Florida. An AST member since 2012, I started volunteering in 2013 after I became an AST-BBST Instructor. Coming up through BBST, I thought educational advocacy was one of the AST’s most important community services. You can’t advance the understanding of the craft until testers have a solid understanding of what already exists. I really wanted to improve our offering and felt the best way was to help set priorities at the board level.
AST Board of Directors
Elections happen every year with roughly half of the 7 person board up for election each year. The election process starts with a call for nominations and then candidates introduce themselves via questions posted to the web. Finally voting takes place during the time of CAST (typically the first week of August) and on the final day of the conference a new board is announced.
As a member-elect you are elected to a position by the sitting board members based, in part, on your preferences. In 2018 during a discussion with existing board members it came up there was a need for someone to take on the Treasurer position. It wasn’t the role I initially wanted (VP of Education was my first choice) but I felt reasonably competent so I accepted.
As with any official board position it’s a starting point for your contributions. I really wanted to focus on education but my fellow AST-BBST instructor Simon Peter (with whom I taught countless classes) wanted the position as well. We quickly both decided it made sense for him to take VP of Education and I take Treasurer. Just like we had done in our teaching we decided it would be fun to collaborate on the many changes we wanted to see in AST-BBST. I had my official role, Simon had his and yet we worked together whenever we could to improve our educational program.
Often when I’m chatting with someone about their regression testing strategy there is an assumption regression is all about repeating the same tests. This is a bit problematic because it ignores an important aspect which testers tend to be good at: focusing on risk. A better way to think of regression testing is it can be applied in two different ways: Procedurally and Risk-Focused.
Procedural Regression Tests
When I speak of procedural I mean a sequence of actions or steps followed in regular order. As I said above this seems to be the primary way people think about regression testing: repetition of the same tests. This extends to the way we think about automating tests as well.
Procedural regression testing can be quite valuable (so far as any single technique can be). The most valuable procedural regression tests are unit tests when applied to our CI system and run regularly. In this way they become a predictable detector of change, which is often why we run regression tests in the first place. (Funny enough automated UI tests are some of the most common procedural regression tests but aren’t the best detectors of change).
The big problem with procedural regression tests are that once an application has passed a test, there is a very low probability of it finding another bug.
Risk-Focused Regression Tests
When I speak of risk-focused I mean testing for the same risks (ways the application might fail) but changing up the individual tests we run. We might create new tests, combine previous tests, alter underlying data or infrastructure to yield new and interesting results.
To increase the probability of finding new bugs we start testing for side effects of the change(s) rather than going for repetition. The most valuable risk focused regression tests are typically done by the individual testers (or developers) who know how to alter their behavior with each pass through the system.
A Combined Approach
Thinking about regression testing in terms of procedural and risk focus allows us to see two complementary approaches that can yield value at different times in our projects. It also gives testers an escape from the burden that comes with repetition while still allowing us to meet our goals.
My yearly tradition has been to summarize the most popular and important (to me) articles I’ve written over the past year along with some reflections and other forward-looking (and likely wrong) statements mixed in.
I joined BloomNation as a Test Automation Engineer in 2018 and in 2019 was promoted to senior role. The promotion was in part my testing contributions and hitting my goals but also the general impact I’ve had on the company outside of that role. During CAST I was telling people I had temporarily switched positions by taking on a Product Manager role until we could fill the vacancy. While temporary, I helped the company continue to deliver on a new core piece of the business and in turn it gave me some new experience and perspective. I plan to write about this experience soon.
Speaking of CAST 2019, this was my first year helping to organize and run a conference of any kind. I plan to write about this experience as well but suffice it to say this was both fascinating and incredibly hard.
I did some speaking in 2019, made my first and second podcast appearance and found what I hope is a sustainable model for supporting TestingConferences.org. I’ve started to become more serious about potential ways to support the things I do. In fact one theme of 2020 might be me figuring out how to balance all the things I want to do with all the things I’ve already committed to doing. Just like every year!
The Five Most-Viewed Articles:
How to debug problems on Mobile Safari – After showing a co-worker how they could debug mobile safari problems on their MacBook, I realized it wasn’t common knowledge. So I helped change that a bit but making it clear how to do it.
Participating in Code Reviews as a Tester – I’ve always liked the concept of helping testers push their technical understanding to reduce both risk and increase confidence. This post was based on a webinar of the same name and includes the slides and a link to watch the webinar.
How I Became an Automation Engineer – A talk based on my personal experiences of becoming an Automation Engineer and what my role looks like. This also became a blog post with references, slides and an embedded video.
It’s Easier to Write about Tooling – Whenever I go to conferences there is a heavy emphasis on what our tooling is. Even in my own writing there can be a heavy emphasis on tools and I think that’s just because it’s easier than writing about the decisions we took and models we made prior to choosing it.
Over 185k page-views in 2019
The first article made it into my top 10 articles over all, which is great. Traffic to this site continues moving up and to the right over time. In 2019 alone I had more than 185k page-views. That’s double the views in one year! Wow!
A Few Other Articles
It was the Creative Web that collapsed – The title of this post is a line from Edward Snowden’s new book, Permanent Record. It’s a great book, you should read it. I reflected on how Surveillance Capitalism is part of the world of the web and how I hope to limit that surveillance on all my sites.
Getting Paid to Learn was a reminder of how important it is to have the company you work for support your professional development. Making promises is one thing but taking action to do it has a positive effect on the people and the company culture.
These are articles were written because I was feeling the flow and when inspiration hits you take it. They turned out well, I’m proud to write consistently about topics that intrigue me and I hope there’s intrigue on the other side too.
The Future is already here
Predicting the future is fun and yet meaningless. But here are the things already on my radar for 2020. So much to consider and so little time:
CAST 2020 is already rolling with early bird ticket sales open.
I’m almost done teaching an AST-BBST Foundations course, my first in a few years and I’m constantly looking at the course with an eye for improvements.
Speaking of improvements, I’m still helping to redesign some aspects of the course AND of course trying to be a treasurer and handle all the financial things.
When it comes to writing I hope to continue the pace of 2x blog posts per month. It’s challenging but doable. More would be great but I want a constant pace to keep myself going.
Despite record setting viewership in 2019, I’m hoping to get better at self branding and sharing which should see page-views increase YOY again. I’ve already updated the newsletter to be less often so I can focus on driving readership for certain articles. I also intend to cross-promote on other blogging and reading platforms to gain viewership. So much to do.
I essentially did 3 presentations / talks in 2019 and I intend to do the same or a few more this year (especially for the online conferences). Hopefully I’ll also get around to making a few more podcast appearances because they are fun to do and fun to share.
Cheers to the rest of 2020! What will you be doing?
During CAST I sat for an interview with Pradeep Soundararajan of Moolya Testing. We talked about a few things mostly on what it meant to be a leader in the testing space. They made a short video on the interview so check it out or read the transcript.
When I think of test leadership, I think of two angles: one is the thought leadership within the industry itself, and in the other is the experience I’m able to impart on my coworkers. So sort of two diverging ideas. One of them is because I’m often the sole tester. It usually means I’m sort of the de facto person that knows testing a bit more than everyone else. It’s like how do you coach and extend and just sort of bring up the level of testing and quality within an organization? Then the other part is like, what are other people doing? What are the things I’m missing? What are, you know, just constantly looking at what the industry and sort of beyond are doing?