TDD and Software Testers

I’ve been following along with the series of conversations with Martin Fowler, Kent Beck and David Heinemeier Hansson (DHH) entitled Is TDD Dead. The whole conversation about what’s good, bad and ugly with test driven development (TDD) is interesting in my role as a software tester and from an overall system / quality perspective. What works, what doesn’t? What do some programmers like about it and what do others fear? Does TDD translate into a better product? Etc.

According to Fowler’s website, part 3 of the series covers

…the various ways in which we get feedback while programming and the role of QA in providing feedback to developers.

The whole series is worth a watch but if you are just interested in TDD and the role it plays when you have software testers (or QA), watch it here:

The three people involved with it have have varying experiences with Fowler having worked for many years with software testers in enterprise software, Beck now working at Facebook where they have few testers (and his own experience with dysfunctional QA) and DHH’s experience running Basecamp. It’s an interesting and relevant discussion because it’s coming from a programmers point of view (programmer testing).  My view says testing is an investigation designed to reveal information about a product. Beck frames it as feedback that builds confidence in the code. I think both views of the software are valuable and those differences in techniques and approaches yield very different ways of viewing quality.

The title “TDD is dead” reminds me of the saying “Test is dead”. Neither of those titles are accurate (they are catchy) but understanding the differences in views can help us when talking to stakeholders who have similar feelings or views. 

Conference Idea – A Bug Advocacy Boot Camp?

If someone asked you to speak at or submit a proposal to speak at a conference do you know what you’d say?

I’d say no thanks for now, but I do have an idea for a tutorial. (This is a not-so-well thought out brainstorm and not a proposal.)

Anyone who’s read my blog of late knows I’m a fan of the Black Box Software Testing courses and anyone who has taken a course knows there’s more material cover than the typical 4 week class can get to. It’s very good material but it’s overwhelming at times. One of the classes, Bug Advocacy, has an immediate real world impact for practitioners. If the class could be sliced up in a way that helped participants understand:

  • Basic concepts (quality, bug workflow) 
  • The value of writing good bug reports for the bug writers (testers, programmers, etc.) 
    • For the Bug Reporters – testers primary work product and they put our credibility on the line 
    • For the organization – Well written bug reports can have an impact on quality and the development lifecycle 
    • Motivating the Bug Fixer
  • How to Write a Good Bug Report 
  • This boot camp is not a supplement to the full Bug Advocacy course

I think it could be a valuable Full Day Tutorial. Not as valuable as taking the full class but still valuable for a professional conference.

The challenges are numerous. This is typically a four week course with six recorded lessons, hundreds of slides, a final written exam and a few assignments that include requiring students to analyze a product and write persuasive, well written bug reports. The class also requires students to join the Open Office project, select a few poorly written bug reports, analyze them and submit re-writes that are better. There’s also a lot of time for peer-review and comparisons of well written and not so well written bugs.

Perhaps by focusing more on the Learning Objectives and less on content with some staged / preplanned simple and complex assignments would work?

This idea is interesting for a number of reasons:

  • It would further my understanding of the material and how to teach it (instructional design) which is part of the reason I became a BBST Instructor (I’d like to one day teach the material on my own). 
  • Bug Advocacy has enough practical material that it would be compelling to students who may not be aware of the BBST classes.
    • Further raise awareness for BBST classes and those who teach it (AST & KFA) 
  • Get more people involved in the open courseware community and potentially lead to better material.

I’d want a number of trial runs with this new instructional design before I applied to a conference. Perhaps after a few classes in my new role as an AST-certified BBST Instructor and then by some willing participants at a full day meet up I might have a better idea of the viability of a boot camped / introduction to Bug Advocacy. Maybe there would be a way to include the eventual Bug Advocacy Workbook in the class or as a way for the students to follow up. (Note the Foundations Workbook has just been released and I have no idea when the Bug Advocacy one will be out.)

Any ideas or advice?

(Thanks to Becky Fiedler who suggested Bug Advocacy as the BBST class most appropriate for a full day conference during a discussion we had during WTST / BBST Domain Testing last month.)

I’m an AST BBST Instructor

That’s right, I completed the trifecta of BBST classes and decided to continue on to being an Instructor. I finished the class in November but just recently got the proof:

Boom! Now comes the hard work – working as an assistant through enough classes until I learn the system and work my way up to a lead instructor role for AST.

Part of my desire for taking the BBST Instructor class is to become more familiar with material taught in the BBST classes, get exposed to new student’s ideas (that will hopefully broaden and challenge my understanding) and learn how to provide important and useful feedback (and help others do the same thing).

Eventually I hope to teach the BBST materials to people I work with and help expose others to the material outside of AST. This was the first step.

State of Testing Survey 2013

Lalit Bhamare a co-founder and editor of Tea Time with Testers magazine contacted me to share a project he and Joel Montvelisky came up with – a State of Testing survey. Lalit said it best when he described why they wanted to undertake this survey:

Some weeks ago my friend, Joel Montvelisky was looking for information to write a post about the advances in the testing world in the last 5-10 years and he realized that there is no centralized set of information that provides visibility into what is happening and what are the trends in the world of testing today. In principle he was looking for something similar to the State of Agile survey that he reviews each year when it goes out, and he was not able to find something that provided such information.

Lalit Bhamare

So Joel contacted me and we decided to turn this into a project i.e. to conduct a State of Testing Survey that will provide a snapshot of our testing reality and help us to capture some of the trends as they shift year by year.

Lalit Bhamare

They want to understand a few things about the testing community (click to see a bigger image):

I think this is a great idea so I signed up to be notified when the survey goes live. I encourage you to do the same by going to Oh and hurry it will only be available for 10 days, starting towards the end of this week!

See you at WTST 2014

The Workshop on Teaching Software Testing, abbreviated as WTST and I think pronounced “what’s it”, is coming up at the end of January 2014 in Melbourne, Florida near the Florida Institute of Technology campus. (You maybe have seen a few of Cem Kaner’s posts.) Just recently my application was approved and I’m excited to be attending as a non-presenting participant.WTST is a LAWST-style workshop which means unlike regular conferences it’s limited to small group of active participants, some who present and everyone is expected to engage in discussions by asking question, providing their experiences and even thoughtfully arguing with the presenter. Discussion is the entire point of the workshop.

This year WTST is focused on designing and teaching advanced courses in software testing, something I’ve had an interest in for a few years now. Teaching software testing was the driving force to me becoming an Instructor for the Association for Software Testing’s BBST classes (which I completed just recently).

This year the workshop will be broken into two sections: The first is the typical weekend-long WTST workshop and the second is the Domain Testing Workshop held for five days immediately after the traditional WTST. I’m excited to be part of a pilot course built around The Domain Testing Workbook (see my previous post for more info) and I imagine it will be a fun exercise in understanding domain testing a little better.

Who else is going? Hopefully I’ll see you there. =)

For more information see

The Domain Testing Workbook is available

Cem Kaner, Sowmya Padmanabhan and Doug Hoffman have a new book called The Domain Testing Workbook. I’d highly recommend picking up a copy or at least adding it to your reading list! This book is not just a deep dive into one test technique but it represents a collective thinking about what software testing is today.

Domain Testing

BBST Test Design was my formal introduction to Domain Testing aka boundary and equivalence class analysis. Domain Testing is often cited as the most popular (or one of the most popular) test techniques in use today. For its part the Test Design course spends a whole week, a full lecture series and at least one assignment introducing and practicing this technique. If you’d like an introduction I recommend the first lecture from the fifth week in the Test Design series in which Cem introduces Domain Testing:

(For more information see the Testing Education Website or YouTube.)

I got the chance to do some reviewing of the workbook and enjoyed how it elaborated on the material I was learning Test Design. Yet it went further by offering layers of detailed examples to work through and is set up to allow the reader to skip around to whatever section or chapter they find interesting.

In an email, Cem described the book:

My impression is that no author has explored their full process for using domain testing. They stop after describing what they see as the most important points. So everyone describes a different part of the elephant and we see a collection of incomplete and often contradictory analyses. Sowmya, Doug and I are trying to describe the whole elephant. (We aren’t describing The One True Elephant of Domain Testing, just the one WE ride when we hunt domain bugs.) That makes it possible for us to solve many types of problems using one coherent mental structure.

Cem Kaner

I’m excited for this book to be turned into a class on domain testing ( and hopefully open sourced) with the rest of BBST. In the meantime pick up the book from Amazon and let me know what you think!

How to Write a Good Bug Report (and be a Bug Advocate)

For the previous BBST classes I made the mistake of not writing about what I learned and the value the classes provided me. To an organization who might consider sponsoring someone (a programmer, analyst…)  or to a fellow tester who is looking for a step in the right direction I have no story to tell, until now.

For Software Testers bug reports are one of our most visible work products – something tangible that comes from our efforts of investigating the system. How well testers write a report can effect how we are perceived (to those who read them) which in turn effects our credibility and influence among those same people (especially programmers). Specifically for development and testing groups, bug reports are important communication mediums (think about the amount of time you spend reading and writing them) and as such an emphasis on understanding bug reports and improving writing skills is important. This is why I took the  Bug Advocacy course through AST and here’s what I learned.

How to Write a Good Bug Report

Writing a bug report seems simple. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, experience but just because something is easy to do doesn’t mean its easy to do well.

So what’s the point of writing a bug report? To get the bug or problem fixed! That’s why the lead author of the material, Cem Kaner, calls it Bug Advocacy – we are advocating for a fix. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written. Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.

How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs.

To “sell” someone on fixing the bugs we need to write a good report. To help us understand the elements of a good report we can use the heuristic RIMGEA:

R – Replicate it. When writing a bug report we should include enough information to get the bug fixed including the steps or conditions necessary to replicate the problem on another machine. The lecture recommends using at least 2 different machines – a powerhouse computer and another less powerful / resource constrained machine. (Virtual environments might work as well.)
I – Isolate it. After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal. We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.
M – Maximize it. Maximizing the problem means doing some follow up testing to see if we can find a more serious problem than the one we were originally going to report. Follow up testing can consist of varying our behavior, our environment, our inputs or even our system configuration. The more serious the problem the more likely (or easier) it will be to convince someone to fix it. Example: We find a problem in OO Impress where it won’t import old PowerPoint files; the application just won’t respond. After some follow up testing we find Impress actually crashes if we use an old PowerPoint file over a certain size.
Maximizing a bug (or follow up activities as it’s also known) is one of the more interesting and time consuming aspects of writing a bug report. There’s quite a bit of detail just in the four categories of variables: vary our behavior, vary our environment, vary our data and vary our system configuration and for those interested Markus Gartner goes a little more in-depth about each in this post.
G – Generalize it. Generalizing the bug means seeing if the problem affects more people in more ways than we originally thought. If the bug was a corner case maybe we can un-corner it, show that it occurs under less extreme conditions. (replicating on a broader range of systems?) Example: Same problem as above in OO Impress where it crashes when we try to import old PowerPoint files over a certain size. We vary the data (input) and find out Impress actually crashes when we try to import any file type over a certain size. Now the problem looks really bad and affects more users than just older PowerPoint users.
E – Externalize it. Switch the focus of the report from the program failing to the stakeholders who will be affected by it and show how it will affect them. (Historical review?) Example: OO Impress crashing when importing any file type of a certain size will stop all users from importing comparable application files (like from PowerPoint). This could prevent new users from downloading and installing the application and even turn off existing users who rely on the compatibility.
A – And say it clearly and dispassionately. Make sure the bug report is easy to understand and that the tone of the report is neutral so as not to offend anyone reading it (perhaps the developer who created the bug) and/or discredit your reputation as the bug reporter (at least in the eye of the reader).

To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the Fails, When Protocol which says describe the failure first and then when that failure happens second. For example in the Bug Advocacy class we looked at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one.

If I had used the protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”.  To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?

How to be a Bug Advocate

We want to make programmers, managers and other stakeholders want to fix the bugs and we can do that by influencing their decision. Writing clear reports can lead to that happening but it’s important to be aware of other things that can influence our ability to become effective Bug Advocates (and effect our ability to write those reports):

  • There are lots of definitions of a software error (so make sure you understand your organizations or teams).
  • There are lots of definitions of quality (so make sure you understand your organizations or teams).
  • Report readers (developer, testers, customers) carry biases with them. Try to minimize the effect these have on them to get the problems fixed.
  • Keep in mind there are lots of reasons why a bug might not get fixed:
    • Time constraints of the people involved
    • Cost of fixing is too prohibitive
    • Trade off between the cost of fixing and adding a feature people like (loss in perceived value due to the problem, an increase in value by adding a new feature customers want, opportunity cost, etc.)
    • Credibility of the bug writer is low in the readers eyes (manager, programmer, etc.)
    • Problem is a corner case (not likely to affect a lot of users)

The value of Bug Advocacy

The value for the bug writer (tester, programmer, analyst, etc) is pretty simple:

  • Experience writing clearer and more effective bug reports
    • Better bug reports can increase the tester’s credibility in the reader’s eyes (programmer)
  • Experience evaluating other people’s bug reports and providing constructive feedback for them to improve
  • Greater understanding of the details / elements involved in the communication medium of bug reports (think biases)
  • Contribute time and effort to an Open Source project (think giving back to the community)
  • Public references to your Open Source work (think for your portfolio)
  • Better understanding in the trade offs between fixing and not fixing problems

The value of the class for an organization thinking of sponsoring an individual:

  • Better written, evaluated and understood bug reports which can lead to a greater efficiency (time saved reading) and understanding of new reports.
  • Increased desire in going back through existing reports, evaluating them and improving them (increasing the likelihood they get addressed).
  • Increased credibility of bug report writers (might help identify and decrease existing biases).
  • Better understanding of organization bias
  • Better understanding of organization needs

I’d like to leave you with one of my favorite quotes from the Bug Advocacy course (courtesy of Cem Kaner):

The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.

To do that you must have credibility, understand the scope of quality for your organization, have experience writing effective bug reports and in general be a bug advocate.

I’m a Bug Advocate

I do advocate for bugs to be fixed but the title comes from passing the Association for Software Testing‘s (AST) Black Box Software Testing (BBST) Bug Advocacy course. The class officially ended in mid-July and it marks the third and final BBST class for me. Together Foundations, Bug Advocacy and Survey of Test Design make up a semester long class that Cem Kaner teaches at Florida Institute of Technology called Software Testing 1. These lectures and classes are, as far as I know, the only college level software testing courses available to anyone that wants to learn about testing – the material is free and the courses are pretty inexpensive.

As with the prior classes I agreed to have my name listed on AST’s graduates website and for fun my certificate of completion is below:

Despite this being the final BBST class my journey with BBST doesn’t end here. I enrolled in the Instructor class for November at the beginning of the year when I enrolled in Test Design. I figured the best way to become more familiar with the material and reinforce what I do know (challenge what I don’t know) is to try to teach it. Worked for Scuba Diving!

Why take a Black Box Software Testing course?

I was recently telling a friend about the BBST Bug Advocacy course I was working on and he asked why I was taking a black box testing course. I think what he meant was why would I take a course on black box testing as opposed to glass box (or white-box) testing?

This is the more thoughtful answer I wish I had given.

I fell into the testing profession. The University I went to taught some technical skills as part of the degree (programming, database design, etc.) but I never learned anything I would consider fundamental to understanding software testing, nothing that helped me deal with testing problems. Since I work in software testing I wanted to learn more about the domain, to better understand it and be able to differentiate between testing and other types of problems (technology, communication, requirements, etc.). It just so happens the courses that focus on the domain of software testing also focus on black box system level testing.

Black box and glass box testing approaches focus on different things. Black box is testing and test design without knowledge of the code. A black box tester will approach testing based on the interactions of the stakeholders with the system’s inputs and outputs. Contrast this to glass box as testing and test design with knowledge of the internal code. The glass box tester will approach testing based on the interactions of the underlying code, as in “does this code do what the programmer intended it to do?”

Testers using either approach can benefit from understanding the domain of software testing. Understanding what oracles are, how they are heuristics, advantages / disadvantages to measurement, how to communicate and write bugs effectively or even how to design appropriate tests affect both the black box and glass box approaches. That’s why someone would take a black box testing course (specifically the BBST series of courses); that’s why I did.

I’m a Black Box Test Designer

More accurately I’ve passed the Association for Software Testing‘s (AST) Black Box Software Testing  (BBST) Test Design class. Test Design is a survey class, students are introduced to 30+ types of tests but there is only enough time to focus on a few of them: risk-based testing, specification-based testing and domain testing. In December I passed the Foundations course and a few days ago I started the Bug Advocacy course (there are 3 courses in the BBST series) which sets me on the path to meet my goal of taking all 3 courses this year. I also signed up for the Instructors course towards the end of the month, wahoo!

I agreed to have my name listed on AST’s graduates website but just for fun here is my certificate of completion:

The Test Design instructors thought I understood enough of the information to pass me. Test Design was like the Foundations class – a lot of reading, labs and collaboration. In the time between Test Design ending and Bug Advocacy starting I cleaned up some of the labs but I haven’t finished. My goal will be to turn these labs (work products) to build out my testing portfolio. Some day I might even make that portfolio public so others can see what I’ve done and perhaps use it as inspiration to build their own.