Sign in Subscribe

How to Write a Good Bug Report (and be a Bug Advocate)

For the previous BBST classes I made the mistake of not writing about what I learned and the value the classes provided me. To an organization who might consider sponsoring someone (a programmer, analyst…)  or to a fellow tester who is looking for a step in the right direction I have no story to tell, until now.

For Software Testers bug reports are one of our most visible work products – something tangible that comes from our efforts of investigating the system. How well testers write a report can effect how we are perceived (to those who read them) which in turn effects our credibility and influence among those same people (especially programmers). Specifically for development and testing groups, bug reports are important communication mediums (think about the amount of time you spend reading and writing them) and as such an emphasis on understanding bug reports and improving writing skills is important. This is why I took the  Bug Advocacy course through AST and here’s what I learned.

How to Write a Good Bug Report

Writing a bug report seems simple. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, experience but just because something is easy to do doesn’t mean its easy to do well.

So what’s the point of writing a bug report? To get the bug or problem fixed! That’s why the lead author of the material, Cem Kaner, calls it Bug Advocacy – we are advocating for a fix. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written. Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.

How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs.

To “sell” someone on fixing the bugs we need to write a good report. To help us understand the elements of a good report we can use the heuristic RIMGEA:

R – Replicate it. When writing a bug report we should include enough information to get the bug fixed including the steps or conditions necessary to replicate the problem on another machine. The lecture recommends using at least 2 different machines – a powerhouse computer and another less powerful / resource constrained machine. (Virtual environments might work as well.)
I – Isolate it. After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal. We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.
M – Maximize it. Maximizing the problem means doing some follow up testing to see if we can find a more serious problem than the one we were originally going to report. Follow up testing can consist of varying our behavior, our environment, our inputs or even our system configuration. The more serious the problem the more likely (or easier) it will be to convince someone to fix it. Example: We find a problem in OO Impress where it won’t import old PowerPoint files; the application just won’t respond. After some follow up testing we find Impress actually crashes if we use an old PowerPoint file over a certain size.
Maximizing a bug (or follow up activities as it’s also known) is one of the more interesting and time consuming aspects of writing a bug report. There’s quite a bit of detail just in the four categories of variables: vary our behavior, vary our environment, vary our data and vary our system configuration and for those interested Markus Gartner goes a little more in-depth about each in this post.
G – Generalize it. Generalizing the bug means seeing if the problem affects more people in more ways than we originally thought. If the bug was a corner case maybe we can un-corner it, show that it occurs under less extreme conditions. (replicating on a broader range of systems?) Example: Same problem as above in OO Impress where it crashes when we try to import old PowerPoint files over a certain size. We vary the data (input) and find out Impress actually crashes when we try to import any file type over a certain size. Now the problem looks really bad and affects more users than just older PowerPoint users.
E – Externalize it. Switch the focus of the report from the program failing to the stakeholders who will be affected by it and show how it will affect them. (Historical review?) Example: OO Impress crashing when importing any file type of a certain size will stop all users from importing comparable application files (like from PowerPoint). This could prevent new users from downloading and installing the application and even turn off existing users who rely on the compatibility.
A – And say it clearly and dispassionately. Make sure the bug report is easy to understand and that the tone of the report is neutral so as not to offend anyone reading it (perhaps the developer who created the bug) and/or discredit your reputation as the bug reporter (at least in the eye of the reader).

To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the Fails, When Protocol which says describe the failure first and then when that failure happens second. For example in the Bug Advocacy class we looked at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one.

If I had used the protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”.  To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?

How to be a Bug Advocate

We want to make programmers, managers and other stakeholders want to fix the bugs and we can do that by influencing their decision. Writing clear reports can lead to that happening but it’s important to be aware of other things that can influence our ability to become effective Bug Advocates (and effect our ability to write those reports):

  • There are lots of definitions of a software error (so make sure you understand your organizations or teams).
  • There are lots of definitions of quality (so make sure you understand your organizations or teams).
  • Report readers (developer, testers, customers) carry biases with them. Try to minimize the effect these have on them to get the problems fixed.
  • Keep in mind there are lots of reasons why a bug might not get fixed:
    • Time constraints of the people involved
    • Cost of fixing is too prohibitive
    • Trade off between the cost of fixing and adding a feature people like (loss in perceived value due to the problem, an increase in value by adding a new feature customers want, opportunity cost, etc.)
    • Credibility of the bug writer is low in the readers eyes (manager, programmer, etc.)
    • Problem is a corner case (not likely to affect a lot of users)

The value of Bug Advocacy

The value for the bug writer (tester, programmer, analyst, etc) is pretty simple:

  • Experience writing clearer and more effective bug reports
    • Better bug reports can increase the tester’s credibility in the reader’s eyes (programmer)
  • Experience evaluating other people’s bug reports and providing constructive feedback for them to improve
  • Greater understanding of the details / elements involved in the communication medium of bug reports (think biases)
  • Contribute time and effort to an Open Source project (think giving back to the community)
  • Public references to your Open Source work (think for your portfolio)
  • Better understanding in the trade offs between fixing and not fixing problems

The value of the class for an organization thinking of sponsoring an individual:

  • Better written, evaluated and understood bug reports which can lead to a greater efficiency (time saved reading) and understanding of new reports.
  • Increased desire in going back through existing reports, evaluating them and improving them (increasing the likelihood they get addressed).
  • Increased credibility of bug report writers (might help identify and decrease existing biases).
  • Better understanding of organization bias
  • Better understanding of organization needs

I’d like to leave you with one of my favorite quotes from the Bug Advocacy course (courtesy of Cem Kaner):

The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.

To do that you must have credibility, understand the scope of quality for your organization, have experience writing effective bug reports and in general be a bug advocate.

Subscribe to Shattered Illusion by Chris Kenst

Sign up now to get access to the library of members-only issues.
Jamie Larson
Subscribe