How to Write a Good Bug Report, use RIMGEN

RIMGEN is an acronym and mnemonic (or memory aid) to help us remember the elements of a good bug report. It can be used to help anyone write a better bug report or review existing bug reports looking to make improvements. 

In general my preference with reporting bugs is to:

  1. Work with a developer to get them fixed, either by mentioning it passing or pairing with them. By far the most effective way to get bugs fixed.
  2. Fix it myself, although most of these are very simple bugs.
  3. Write a bug report. These are good instructions for that process.

How to Write a Good Bug Report

Writing a bug report seems simple on the surface. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, practice. Although we’ve all seen those bug reports come in without much detail or context and guess what? They don’t get fixed. Just because something is easy to do doesn’t mean it is easy to do well.

We’re writing a good bug report to get a problem fixed. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written (mention it in passing or through slack). Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.

How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs. To “sell” someone on fixing the bugs we need to write a good report.

To help us understand the elements of a good report we can use the heuristic RIMGEN.

RIMGEN:

R – Replicate it

When writing a bug report we should include enough information to get the bug fixed including the steps or conditions necessary to replicate the problem on another machine. The lecture recommends using at least 2 different machines – a powerhouse computer and another less powerful / resource constrained machine. (Virtual environments might work as well.)

I – Isolate it

After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal.

We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.

M – Maximize it

Maximizing the problem means doing some follow up testing to see if we can find a more serious problem than the one we were originally going to report. Follow up testing can consist of varying our behavior, our environment, our inputs or even our system configuration. The more serious the problem the more likely (or easier) it will be to convince someone to fix it. Example: We find a problem in OO Impress where it won’t import old PowerPoint files; the application just won’t respond. After some follow up testing we find Impress actually crashes if we use an old PowerPoint file over a certain size.

Maximizing a bug (or follow up activities as it’s also known) is one of the more interesting and time consuming aspects of writing a bug report. There’s quite a bit of detail just in the four categories of variables: vary our behavior, vary our environment, vary our data and vary our system configuration and for those interested Markus Gartner goes a little more in-depth about each in this post.

G – Generalize it

Generalizing the bug means seeing if the problem affects more people in more ways than we originally thought. If the bug was a corner case maybe we can un-corner it, show that it occurs under less extreme conditions. (replicating on a broader range of systems?) Example: Same problem as above in OO Impress where it crashes when we try to import old PowerPoint files over a certain size. We vary the data (input) and find out Impress actually crashes when we try to import any file type over a certain size. Now the problem looks really bad and affects more users than just older PowerPoint users.

E – Externalize it

Switch the focus of the report from the program failing to the stakeholders who will be affected by it and show how it will affect them. (Historical review?) Example: OO Impress crashing when importing any file type of a certain size will stop all users from importing comparable application files (like from PowerPoint). This could prevent new users from downloading and installing the application and even turn off existing users who rely on the compatibility.

N – Neutral Tone

Make sure the bug report is easy to understand and that the tone of the report is neutral so as not to offend anyone reading it (perhaps the developer who created the bug) and/or discredit your reputation as the bug reporter (at least in the eye of the reader).

BUG TITLES AND SUMMARIES

To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the “Fails, When Protocol”. “Fails, When” says describe the failure first and then when that failure happens second.

For example in Bug Advocacy we look at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one. If I had used the “Fails, When” protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”.

To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?

A Few Other Things

  • Looking for something a little more portable? Checkout my Bug Reporting Guidelines document.
  • This is an update of my previous post “How to Write a Good Bug Report (and be a Bug Advocate)” with some slight alterations.
  • Design vs coding bugs are slightly different and have a different emphasis using RIMGEN. I hope to address this later in a separate article.
  • Advocating for a bug to be fixed is not an invitation to be sleazy or aggressive in your reporting. We are trying to influence a decision based on our research and understanding, not trying to ruin our credibility by pushing something the business has no desire to fix.
  • RIMGEN is the updated version of RIMGEA

Facebook Ad’s Massive Design Bug

In mid-September ProPublica published an article proving Facebook’s advertising system helped them market to people who expressed interest in radical and racial topics:

“Want to market Nazi memorabilia, or recruit marchers for a far-right rally? Facebook’s self-service ad-buying platform had the right audience for you.

Until this week, when we asked Facebook about it, the world’s largest social network enabled advertisers to direct their pitches to the news feeds of almost 2,300 people who expressed interest in the topics of “Jew hater,” “How to burn jews,” or, “History of ‘why jews ruin the world.’” Published September 14th, 2017

Turns out it was technically possible to use Facebook ads to target users claiming to be anti-semitic. This same system could perhaps be used by Russians to target American Voters but that’s a whole other thing. Facebook’s ad system allows advertisers to type categories of users they’d like to target as an input field and then an automated matching system will check those user supplied categories (full or parital) with ones Facebook already had like this:

Aside from the moral and ethical problems of targeting users and groups this way, from a software perspective we might classify this as a design bug. Unlike a coding bug where the program behaves in a way the programmer didn’t intend, a design bug is when the program behaves in a way the programmer did intend but stakeholders don’t like it. In this example Facebook’s ad system was behaving as it was programmed but in a way that was HIGHLY questionable to many stakeholders like advertisers, advertisees, journalists and now federal authorities.

Design Bugs

Design bugs are some of the most common types of bugs developers (testers) find. They are are often caught by questioning the system, approach, design, requirements, et .al in a holistic way. In cross functional and agile teams I’ve found this is easier because you can take a step back to look at the bigger picture with many of the original decision makers. Going through the implications of those decisions as a mental exercise will often expose a bug or two that can be then addressed in the near future.

To help in questioning design bugs, try to:

  • Find a set of scenarios or circumstances that showcase your concerns
  • Make those concerns as costly or annoying as possible
  • Find places outside of the program where users will be impacted

For those familiar with the RIMGEN bug reporting mnemonic, you might notice some similarities, although with a different emphasis.

We (all software developers) introduce design bugs all of the time with the justification of it “works as designed”. Most times the design of a system is’t so damaging. Sometimes it is incredibly.[highlight]This lesson seems worth keeping in mind. [/highlight]

Other Things:

  • You can see Facebook’s response here.
  • Benedict Evans suggested this was more of an exploit and programmers (I say all software developers) should consider what happens when users are evil. This is probably a valuable approach. I often take the approach of “anything you allow me to do I will do”. In other words, give me enough room to hang myself and I will.

Recognizing a problem in eBay’s iPad app

I’ve been buying and selling things on eBay for more than a decade. Naturally in the last few years I’ve spent more time on the iPhone app but for some reason I wasn’t using the iPad app. I figured it was time, so I installed the eBay app, logged into my account and went to the selling page to view my active auctions. The photo below is what I saw:

eBay iPad

Immediately I recognized a problem.

Do you see it?

On this selling page there is nothing to tell me how many bids each auction item has, assuming they have any, or if any auction will be sold. Another way to look at it: I can’t tell which auction items are going to make me money!

Confusing matters is the use of black and red colors for the current prices. Does red mean the item won’t sell? Does black mean it will? No, that doesn’t appear to be the logic. Only two of my items have bids – the third item (shown in black) and the last auction item (also shown in black). So why do the non-selling items have colors of red and black? Like I said, confusing.

Identifying problems like this can seem obvious when you have sufficient experience with a product (or someone explains it) but even without help there are ways to identify and evaluate problems such as this. All we have to do is find an oracle (a way to recognize a problem) and do some testing (perform an investigation). When reporting and evaluating problems like this I like to use the collection of consistency oracles by James Bach and Michael Bolton. You should be able to follow along fine with the rest of this essay if you’ve never seen list but it’s worth a read.

After evaluating the list of oracles I want to call attention to the ones I think help highlight the problems with the iPad app. The order below is based on my observations of the problem as I came across them. It just happens the evidence becomes stronger and more convincing as we work through the list.

  1. Inconsistent with user’s desires
  2. Inconsistent with purpose
  3. Inconsistent within product

Inconsistent with user’s desires

I think it’s reasonable to expect eBay sellers with current listings, will want to know how many of those listings have bids and if they’ve met the minimum criteria for completing the sale including reserve amounts, number of bids, etc. In fact I’d bet that’s one of the most important pieces of information they’d want to see because it’s what I wanted to see.

Wait a minute, can’t a user click on every single auction item and view more details? Yes. Assuming the user is like me and only selling a few items it’s probably a fine work-around. However, what if you’ve listed a hundred or a thousand items like eBay PowerSellers and businesses do? Do you think it’s reasonable to expect them to click on every single auction? I don’t; it defeats the purpose of the selling view.

User desires can be a hard argument to make on its own. Unless we knew eBay valued this as a high-risk area or we found it affected a large number of users, we probably need to do more research to back up our argument.

Inconsistent with purpose

I think the explicit purpose of the selling page is to help eBay sellers monitor auction items and complete sales. In fact this is what I use it for. Typically with auction durations of more than one day I will glance at each listing once per day by going to the selling page (pictured above). When the auction duration gets to be under 24 hours I will visit the page far more frequently.

Additionally by using different colors for the price of an auction that will sell vs. one that won’t, eBay can quickly identify the status of each listing item. For example if green meant an auction would sell and red meant it wouldn’t, I could easily scan through my selling page and implicitly understand how things were going. From there I could make adjustments if needed.

Since the iPad app shows the same color for auctions that will and won’t sell and because I can’t monitor my auction sales at a distance I think the sellers page is inconsistent with it’s purpose. Given my experience with the eBay product as a whole I also know it’s inconsistent with the larger eBay product.

So far we’ve covered two inconsistencies, two ways we can identify the problems I came across. In both I’m arguing, based on my experience, I understand what a user wants (as a user myself) and I understand the purpose of the product. If both of these inconsistencies sound similar that’s ok because in this case they happen to. They won’t always.

Other than my experience I don’t have much data to back up our argument. I could do more research on the product, look for claims, marketing materials, and interviews with experts that would add more evidence to our argument but let’s focus on the last inconsistency.

Inconsistent within product

As I mentioned before I’m a long time eBay user through eBay.com and the iPhone app. Though eBay.com and mobile apps may seem like separate products they are in fact different ways of gaining access (think distribution channel) to the same product, the eBay platform. This means we can look to those channels when thinking about product consistency.

In my mind, even though the iPad app fails to fulfill its users desires and purpose it is most obviously inconsistent with other aspects of the eBay product. Here’s the same seller’s view on the eBay iPhone app:

eBay iPhone seller view 1

eBay iPhone seller view 2

Observe any differences?

The iPhone version shows a similar layout but with a few important differences:

  1. The iPhone app lists the number of bids on each listing right away. I don’t have to go to any other detailed view to get this information (also re-affirms it’s importance).
  2. The current price is color coded in a way that fits with our normal assumptions – green means a listing is going to sell and red means it won’t. This makes it far easier for us to monitor our current auctions at a glance and it fulfills it’s purpose better than the iPad app.

If we were to compare either mobile app to eBay.com, we’d see the sellers page is only consistent with the iPhone app. This is the strongest evidence we’ve found that points to problems with the iPad app. It’s a far more credible argument than the others because it seems likely eBay would want it’s iPad channel consistent with its others (eBay.com and iPhone app). Without doing much additional research we’ve found a way to explain to eBay stakeholders that these problems exist and are bad.

Closing Thoughts

Usually I report problems so they get fixed. Yet identifying, evaluating and then describing a problem in such a way it convinces someone else of its importance isn’t as simple as you might initially think. If I did this well I’ve convinced you, the reader, there’s a problem with the selling page of the iPad app.

Now all I have to do is file the bug with eBay and delete my eBay iPad app. Why delete the app? Well I’ve been trying to sell things on eBay for the past month and given its problems there’s very little value in me using it.

References

How to Write a Good Bug Report (and be a Bug Advocate)

For the previous BBST classes I made the mistake of not writing about what I learned and the value the classes provided me. To an organization who might consider sponsoring someone (a programmer, analyst…)  or to a fellow tester who is looking for a step in the right direction I have no story to tell, until now.

For Software Testers bug reports are one of our most visible work products – something tangible that comes from our efforts of investigating the system. How well testers write a report can effect how we are perceived (to those who read them) which in turn effects our credibility and influence among those same people (especially programmers). Specifically for development and testing groups, bug reports are important communication mediums (think about the amount of time you spend reading and writing them) and as such an emphasis on understanding bug reports and improving writing skills is important. This is why I took the  Bug Advocacy course through AST and here’s what I learned.

How to Write a Good Bug Report

Writing a bug report seems simple. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, experience but just because something is easy to do doesn’t mean its easy to do well.

So what’s the point of writing a bug report? To get the bug or problem fixed! That’s why the lead author of the material, Cem Kaner, calls it Bug Advocacy – we are advocating for a fix. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written. Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.

How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs.

To “sell” someone on fixing the bugs we need to write a good report. To help us understand the elements of a good report we can use the heuristic RIMGEA:

R – Replicate it. When writing a bug report we should include enough information to get the bug fixed including the steps or conditions necessary to replicate the problem on another machine. The lecture recommends using at least 2 different machines – a powerhouse computer and another less powerful / resource constrained machine. (Virtual environments might work as well.)
I – Isolate it. After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal. We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.
M – Maximize it. Maximizing the problem means doing some follow up testing to see if we can find a more serious problem than the one we were originally going to report. Follow up testing can consist of varying our behavior, our environment, our inputs or even our system configuration. The more serious the problem the more likely (or easier) it will be to convince someone to fix it. Example: We find a problem in OO Impress where it won’t import old PowerPoint files; the application just won’t respond. After some follow up testing we find Impress actually crashes if we use an old PowerPoint file over a certain size.
Maximizing a bug (or follow up activities as it’s also known) is one of the more interesting and time consuming aspects of writing a bug report. There’s quite a bit of detail just in the four categories of variables: vary our behavior, vary our environment, vary our data and vary our system configuration and for those interested Markus Gartner goes a little more in-depth about each in this post.
G – Generalize it. Generalizing the bug means seeing if the problem affects more people in more ways than we originally thought. If the bug was a corner case maybe we can un-corner it, show that it occurs under less extreme conditions. (replicating on a broader range of systems?) Example: Same problem as above in OO Impress where it crashes when we try to import old PowerPoint files over a certain size. We vary the data (input) and find out Impress actually crashes when we try to import any file type over a certain size. Now the problem looks really bad and affects more users than just older PowerPoint users.
E – Externalize it. Switch the focus of the report from the program failing to the stakeholders who will be affected by it and show how it will affect them. (Historical review?) Example: OO Impress crashing when importing any file type of a certain size will stop all users from importing comparable application files (like from PowerPoint). This could prevent new users from downloading and installing the application and even turn off existing users who rely on the compatibility.
A – And say it clearly and dispassionately. Make sure the bug report is easy to understand and that the tone of the report is neutral so as not to offend anyone reading it (perhaps the developer who created the bug) and/or discredit your reputation as the bug reporter (at least in the eye of the reader).

To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the Fails, When Protocol which says describe the failure first and then when that failure happens second. For example in the Bug Advocacy class we looked at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one.

If I had used the protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”.  To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?

How to be a Bug Advocate

We want to make programmers, managers and other stakeholders want to fix the bugs and we can do that by influencing their decision. Writing clear reports can lead to that happening but it’s important to be aware of other things that can influence our ability to become effective Bug Advocates (and effect our ability to write those reports):

  • There are lots of definitions of a software error (so make sure you understand your organizations or teams).
  • There are lots of definitions of quality (so make sure you understand your organizations or teams).
  • Report readers (developer, testers, customers) carry biases with them. Try to minimize the effect these have on them to get the problems fixed.
  • Keep in mind there are lots of reasons why a bug might not get fixed:
    • Time constraints of the people involved
    • Cost of fixing is too prohibitive
    • Trade off between the cost of fixing and adding a feature people like (loss in perceived value due to the problem, an increase in value by adding a new feature customers want, opportunity cost, etc.)
    • Credibility of the bug writer is low in the readers eyes (manager, programmer, etc.)
    • Problem is a corner case (not likely to affect a lot of users)

The value of Bug Advocacy

The value for the bug writer (tester, programmer, analyst, etc) is pretty simple:

  • Experience writing clearer and more effective bug reports
    • Better bug reports can increase the tester’s credibility in the reader’s eyes (programmer)
  • Experience evaluating other people’s bug reports and providing constructive feedback for them to improve
  • Greater understanding of the details / elements involved in the communication medium of bug reports (think biases)
  • Contribute time and effort to an Open Source project (think giving back to the community)
  • Public references to your Open Source work (think for your portfolio)
  • Better understanding in the trade offs between fixing and not fixing problems

The value of the class for an organization thinking of sponsoring an individual:

  • Better written, evaluated and understood bug reports which can lead to a greater efficiency (time saved reading) and understanding of new reports.
  • Increased desire in going back through existing reports, evaluating them and improving them (increasing the likelihood they get addressed).
  • Increased credibility of bug report writers (might help identify and decrease existing biases).
  • Better understanding of organization bias
  • Better understanding of organization needs

I’d like to leave you with one of my favorite quotes from the Bug Advocacy course (courtesy of Cem Kaner):

The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the right bugs fixed.

To do that you must have credibility, understand the scope of quality for your organization, have experience writing effective bug reports and in general be a bug advocate.

What I’ve been up to lately

Things have been busy in the last month or so and I felt like sharing what I’ve been up to lately. Most of it revolves around software testing:

April saw the start of Dan Ariely’s A Beginners Guide to Irrational Behavior class on Coursera. I knew I had the BBST course coming up so I didn’t commit much time to the class other than watching the video lectures and doing the video quizzes. There are many aspects of irrational behavior that affect what we do in software development and testing – I’d like to write a more in-depth article about that in the future.

On the 14th of April I started the BBST Test Design course and completed it on May 8th. For those who have never taken a BBST class before they are incredibly intense month long courses. The course breaks a single calendar week into 2 class weeks – one week with 4 days, and a shorter week with 3 days and each week requires about 10-15 hours of work in order to do the readings, labs and work on the exam. The class is done but I still don’t know if I’ve passed; regardless I learned a lot.

On April 19th I joined the NRG Global Online test competition. My last post was a reflection on how well I thought I did and despite my low perception, my team ended up winning part of the competition!

I went to STPcon 2013 at the end of April in San Diego where I met up with a few Miagi-Do’ers, met some other testers I’d heard from in the twitter-verse or blog-o-sphere and learned a few things. I’m planning to write an experience report and post it either here or on the newly formed Miagi-Do blog. I think it might apply a little more here but I don’t know how it will turn out because I haven’t written it.

During the Test Design course I picked up on Test Design being the last of the 3 BBST courses and there being 3 more courses – Domain testing, spec-based testing and scenario-based testing listed in Cem Kaner’s diagram. I asked Cem about the domain testing course over twitter and he kindly sent me an email with a draft domain testing workbook which I plan to review – right after I email him back and telling him when.

May 8th through 10th I participated in the Rapid Testing Intensive Online #2 as a peer reviewer. It was fun to sit on the other side and provide some feedback to the students on their work although I would have been more effective if I was able to do the assignment as the students were – I just couldn’t take the time off work. Nevertheless I found participating as a peer reviewer to have its own unique challenges as I interacted with other testers and tried to answer their questions. In the RTIO there’s a ton of material and references coming at the students so it helps to interact and help others.

May 16th I signed up for the BBST Bug Advocacy class that takes place in June. One of my year end goals is to complete all 3 BBST courses and then pursue BBST Instructor so I can help others. In fact as I was writing this I signed up for the BBST Instructors course in October!

Lastly I’m looking for a cheap / free place to host a public Rapid Software Testing course with Paul Holland in the Los Angeles area. Anyone know of a place that can fit 20 people comfortably?

Wow I’ve got a lot to do…