For Software Testers bug reports are one of our most visible work products – something tangible that comes from our efforts of investigating the system. How well testers write a report can effect how we are perceived (to those who read them) which in turn effects our credibility and influence among those same people (especially programmers). Specifically for development and testing groups, bug reports are important communication mediums (think about the amount of time you spend reading and writing them) and as such an emphasis on understanding bug reports and improving writing skills is important. This is why I took the Bug Advocacy course through AST and here’s what I learned.
How to Write a Good Bug Report
Writing a bug report seems simple. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, experience but just because something is easy to do doesn’t mean its easy to do well.
So what’s the point of writing a bug report? To get the bug or problem fixed! That’s why the lead author of the material, Cem Kaner, calls it Bug Advocacy – we are advocating for a fix. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written. Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.
How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs.
To “sell” someone on fixing the bugs we need to write a good report. To help us understand the elements of a good report we can use the heuristic RIMGEA:
I – Isolate it. After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal. We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.
To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the Fails, When Protocol which says describe the failure first and then when that failure happens second. For example in the Bug Advocacy class we looked at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one.
If I had used the protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”. To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?
How to be a Bug Advocate
We want to make programmers, managers and other stakeholders want to fix the bugs and we can do that by influencing their decision. Writing clear reports can lead to that happening but it’s important to be aware of other things that can influence our ability to become effective Bug Advocates (and effect our ability to write those reports):
- There are lots of definitions of a software error (so make sure you understand your organizations or teams).
- There are lots of definitions of quality (so make sure you understand your organizations or teams).
- Report readers (developer, testers, customers) carry biases with them. Try to minimize the effect these have on them to get the problems fixed.
- Keep in mind there are lots of reasons why a bug might not get fixed:
- Time constraints of the people involved
- Cost of fixing is too prohibitive
- Trade off between the cost of fixing and adding a feature people like (loss in perceived value due to the problem, an increase in value by adding a new feature customers want, opportunity cost, etc.)
- Credibility of the bug writer is low in the readers eyes (manager, programmer, etc.)
- Problem is a corner case (not likely to affect a lot of users)
The value of Bug Advocacy
The value for the bug writer (tester, programmer, analyst, etc) is pretty simple:
- Experience writing clearer and more effective bug reports
- Better bug reports can increase the tester’s credibility in the reader’s eyes (programmer)
- Experience evaluating other people’s bug reports and providing constructive feedback for them to improve
- Greater understanding of the details / elements involved in the communication medium of bug reports (think biases)
- Contribute time and effort to an Open Source project (think giving back to the community)
- Public references to your Open Source work (think for your portfolio)
- Better understanding in the trade offs between fixing and not fixing problems
The value of the class for an organization thinking of sponsoring an individual:
- Better written, evaluated and understood bug reports which can lead to a greater efficiency (time saved reading) and understanding of new reports.
- Increased desire in going back through existing reports, evaluating them and improving them (increasing the likelihood they get addressed).
- Increased credibility of bug report writers (might help identify and decrease existing biases).
- Better understanding of organization bias
- Better understanding of organization needs
I’d like to leave you with one of my favorite quotes from the Bug Advocacy course (courtesy of Cem Kaner):
The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.
To do that you must have credibility, understand the scope of quality for your organization, have experience writing effective bug reports and in general be a bug advocate.