Good Tests vs Bad Tests and why you shouldn’t repeat them

A little rant on this concept of Good Tests vs Bad Tests and whether a good test (case) is a repeatable one.

Here’s the imperfect transcript:

Hi everyone, Chris Kenst here.

So my topic today is I want to talk about the difference between a good test and a bad test.

There’re two reasons that I bring this up today:

  • One I saw yet another article talking about how to create effective test cases and then going on to say they should be repeatable, which is of course, wrong your test should not be repeatable.
  • Two I have been asking this question as I hire for my third software tester here at Promenade group.

I am hiring somebody that’s going to be a senior tester. And I think one thing that a senior-level person should be able to do is differentiate between what makes a good test good and a bad test bad.

But so far in the, I want to say 15 or so, maybe 20 at this point candidates that I have asked this over email, very few have been actually able to describe what makes a good test and what makes a bad test. Most candidates seem to fall into this trap of what I saw in this article which is just bad advice where every test is a scenario regardless of what it is that you’re doing.

And this strikes me as very odd because it’s deeply violates this understanding of providing value.

So if you were to hire somebody to do a job and regardless of how the job changed over time, they were to do it the exact same regardless of changing circumstances, you would think that’s a bad candidate. Somebody wouldn’t want to hire.

It’s the same thing with test design. A good test versus a bad test is about value and it’s about focusing on what that test is going to do: its Mission. And so you can’t have the same approach to every kind of test because that means you’re not providing value. You’re not actually aiming to achieve your mission with the testing.

So if you see advice, or you read advice about good test versus bad test and it’s like you should make sure that all your tests are repeatable that’s just not the case.

For example, with boundary analysis and equivalence class partitioning, you don’t want repeatable test because they’re not built to be repeatable. Sure you could repeat them but now that test is no longer very useful and it’s probably not worth running again.

So it’s really challenging.

I will probably write an article about this too but if you’re ever thinking about how to write good test versus bad test, think about the value and think about the mission of the test itself. Then, try and think of all the different types of attributes that might make something valuable now, and something less valuable.

And then focus, on those things that deliver value, whether their power, whether the repeatability, whether they’re easier for coverage, for understanding that, because the better that we get as a community and better, we become better at testing will see that there are lots of different ways we can run tests.

And the more we understand that the more skilled and knowledgeable about our craft would become and this other kind of cool thing is that you can really set yourself apart from other people who may be interviewing.

If you can easily tell someone what a good test isn’t a bad test. If you are looking at someone’s test and going these are all scenario test you built the same thing just over and over again. How come you’re not varying anything?

All of a sudden you are in a much better position, out of all the other candidates because you can easily differentiate your work from their work and you can tie it to value.

So, that’s my rant today about good test versus bad test.

Have a great day, everyone.

Hiring a Software Tester, an Analysis

In May of 2020, back when Promenade Group was still called BloomNation, I opened a job posting for a Software Test Engineer. This was to be the first of many test positions we eventually hire for. After going through the whole process of hiring a software tester, I thought it would be useful to analyze the applicant data with the idea of learning something about how I hire and about the applicants who applied.

About the data

Some of this data was collected through our recruiting system and some was manually entered in by me. I spent a good deal of time crunching through raw data in Excel, then coming up with new questions and going back to find more data. Some of the data wasn’t captured at all and so I made guesses / assumptions. Specifically I did this for the applicants location and gender. I don’t hire based on gender, but I was curious to see how this might have effected the final outcome. Despite having 142 submissions, I ended up pulling data on only 107 resumes.


It was the Creative Web that collapsed

In his book Permanent Record, Edward Snowden suggests the evolution of the internet has gone from a community without border or limit where “anonymity-through-polyonym produced more truth than falsehood” to one that is unrecognizable today. Unrecognizable, in part, due to the loss of individual websites shuttered by the promise of convenience and replaced with eCommerce platforms that lend themselves to surveillance capitalism.

He goes on to say:

“Now, it was the creative web that collapsed, as countless beautiful, difficult, individualistic websites were shuttered. The promise of convenience led people to exchange their personal sites—which demanded constant and laborious upkeep—for a Facebook page and a Gmail account. The appearance of ownership was easy to mistake for the reality of it. Few of us understood it at the time, but none of the things that we’d go on to share would belong to us anymore. ”

Edward Snowden, Permanent Record

It may have been a conscious choice to shutter individual websites in favor of something simpler (like publishing on a writing platform or even Facebook) but it’s also been a conscious choice that I haven’t.

Here and elsewhere I’d rather be in charge of my own personal sites than use someone else’s platform despite the potential ease and built-in audience. I recongize in exchange for putting up with this pain I get my own little piece of the web where surveillance is minimal (the only tracking here is through Google Analytics) and I get the freedom to do what I want and the opportunity to learn from the ownership of it all. Permanent Record is just one resource on learning what it means to minimize surveillance here and elsewhere.

More than anything else over the years it’s been this opportunity to learn that keeps me running my part of the creative web. For many this might sound like a headache but domain management, DNS, caching, SSL, website performance, SEO, uptime, etc. etc. are all opportunities to better understand the various parts of the internet and to become a more knowledgable consumer.

It’s also not uncommon for these duties to fall to the engineering team of a small company or startup and the more familiar you are, the better you can contribute.

Running my own beautiful, difficult, individualistic website is something I grew into over time and what I recommend others do as well. I don’t mind the difficultly given the upsides of what you can learn and doing my small part in limiting surveillance capitalism.

Appropriate Test Documentation & Formatting

The Question

Recently in an online forum a tester person asked:

Does someone have a simple example of test case (excel sheet) format? I am the only one tester in my company and we are trying to arrange the test documentation. Any advice or example will be useful.

I wish someone had given me this advice when I first started out:

Be careful about using someone else’s formatting and/or templates. If you don’t know what you are doing, you might be inheriting their issues and misunderstandings. It’s usually best to understand what you want to accomplish and then find a solution for yourself. Especially if it’s something simple like using Excel for documentation. Tables and matrices are great for organizing tests and don’t cost much money or time to make.

Appropriate Test Documentation

The appropriate documentation for your tests (or test cases) will depend to a large degree on the test technique and approach you use. Test Case Management systems (and similar systems) are often designed for functional or scenarios tests where the format is similar and you can list out multiple steps. Other test techniques such as Path Analysis, Domain Testing and even Risk Based testing can look very different when they are well designed. The same can be said when you decide to take an Exploratory approach to your testing where you are likely to write Exploratory Charters. Cramming all of these different techniques into one system usually isn’t the best idea.

The Smallest Amount

Focus on the smallest amount of documentation you need to do your job well. You have a finite amount of time and if you spend too much of it creating documentation you will have less time to do the work. If you are writing tests for other humans to read and understand, they should be written in a way that communicates this well such as using visuals (mind maps, diagrams, etc.) and summaries. Similarly if you are writing the tests for computers to read and understand they can (need to be) much more granular and step by step.

I’d wager most experienced engineers have gone through this process of: trying to document as much as possible but by the time you are done you didn’t do enough testing and the documentation isn’t done either. Maybe you put it all into a test case management system that no one else cares to read because it’s in this weird format that probably only makes sense to you.

Staying Agile

Focusing on the appropriate documentation for your tests and then creating the smallest amount you need to do your job well also allows you to have agility. Agile is all about being able to adjust to changes as they come in. Quick changes in direction aren’t kind on large amounts of pre-scripted documentation and/or can result in lost work.

I wish this approach was more obvious but a quick google search will reveal it’s not the dominate theory. However, if you read this and take the advice you’ll be learning from my mistakes. Focus on using the smallest amount of documentation you need to be successful and making sure that documentation is appropriate for your test choices will give you agility and future proofing even as you scale your team size and move into the future.

How to Write a Good Bug Report, use RIMGEN

RIMGEN is an acronym and mnemonic (or memory aid) to help us remember the elements of a good bug report. It can be used to help anyone write a better bug report or review existing bug reports looking to make improvements. 

In general my preference with reporting bugs is to:

  1. Work with a developer to get them fixed, either by mentioning it passing or pairing with them. By far the most effective way to get bugs fixed.
  2. Fix it myself, although most of these are very simple bugs.
  3. Write a bug report. These are good instructions for that process.

How to Write a Good Bug Report

Writing a bug report seems simple on the surface. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, practice. Although we’ve all seen those bug reports come in without much detail or context and guess what? They don’t get fixed. Just because something is easy to do doesn’t mean it is easy to do well.

We’re writing a good bug report to get a problem fixed. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written (mention it in passing or through slack). Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.

How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs. To “sell” someone on fixing the bugs we need to write a good report.

To help us understand the elements of a good report we can use the heuristic RIMGEN.


R – Replicate it

When writing a bug report we should include enough information to get the bug fixed including the steps or conditions necessary to replicate the problem on another machine. The lecture recommends using at least 2 different machines – a powerhouse computer and another less powerful / resource constrained machine. (Virtual environments might work as well.)

I – Isolate it

After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal.

We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.

M – Maximize it

Maximizing the problem means doing some follow up testing to see if we can find a more serious problem than the one we were originally going to report. Follow up testing can consist of varying our behavior, our environment, our inputs or even our system configuration. The more serious the problem the more likely (or easier) it will be to convince someone to fix it. Example: We find a problem in OO Impress where it won’t import old PowerPoint files; the application just won’t respond. After some follow up testing we find Impress actually crashes if we use an old PowerPoint file over a certain size.

Maximizing a bug (or follow up activities as it’s also known) is one of the more interesting and time consuming aspects of writing a bug report. There’s quite a bit of detail just in the four categories of variables: vary our behavior, vary our environment, vary our data and vary our system configuration and for those interested Markus Gartner goes a little more in-depth about each in this post.

G – Generalize it

Generalizing the bug means seeing if the problem affects more people in more ways than we originally thought. If the bug was a corner case maybe we can un-corner it, show that it occurs under less extreme conditions. (replicating on a broader range of systems?) Example: Same problem as above in OO Impress where it crashes when we try to import old PowerPoint files over a certain size. We vary the data (input) and find out Impress actually crashes when we try to import any file type over a certain size. Now the problem looks really bad and affects more users than just older PowerPoint users.

E – Externalize it

Switch the focus of the report from the program failing to the stakeholders who will be affected by it and show how it will affect them. (Historical review?) Example: OO Impress crashing when importing any file type of a certain size will stop all users from importing comparable application files (like from PowerPoint). This could prevent new users from downloading and installing the application and even turn off existing users who rely on the compatibility.

N – Neutral Tone

Make sure the bug report is easy to understand and that the tone of the report is neutral so as not to offend anyone reading it (perhaps the developer who created the bug) and/or discredit your reputation as the bug reporter (at least in the eye of the reader).


To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the “Fails, When Protocol”. “Fails, When” says describe the failure first and then when that failure happens second.

For example in Bug Advocacy we look at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one. If I had used the “Fails, When” protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”.

To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?

A Few Other Things

  • Looking for something a little more portable? Checkout my Bug Reporting Guidelines document.
  • This is an update of my previous post “How to Write a Good Bug Report (and be a Bug Advocate)” with some slight alterations.
  • Design vs coding bugs are slightly different and have a different emphasis using RIMGEN. I hope to address this later in a separate article.
  • Advocating for a bug to be fixed is not an invitation to be sleazy or aggressive in your reporting. We are trying to influence a decision based on our research and understanding, not trying to ruin our credibility by pushing something the business has no desire to fix.
  • RIMGEN is the updated version of RIMGEA

Being a Distributed Tester was hard

As a new father, being able to work a few days per week from home has been a great way to help adjust to our wonderful new addition and the demands that go along with him. Days in the office are great for face time and collaboration and days at home are for getting shit done and helping with the kid:

In the past I’ve worked for a few companies where the development teams were distributed across the US, India, Ireland, Vietnam and the Netherlands. I took advantage by working from locations all over the world. It was (and is) a very freeing experience.

Yet working in distributed environments (where everyone works remotely) is not without its challenges. As Alister Scott pointed out in his talk on “The future of testing is distributed” things like loneliness, lack of accountability, focus, and work/life balance all become more of an challenge in addition to the difficulties around communication. I was fairly successful at adjusting to most of those challenges except for the difficulties around communication.

Perhaps the problem was both companies I worked for were not fully dispersed, only the development teams were. We had tools to help alleviate communication problems like Slack, Skype and daily standup meetings through video (like Google Hangouts). They were important but only partially lessened the communication problems. In both cases the development and product information was shared disproportionately among team members. I usually felt like I last to know (except I wasn’t the only one.) Sometimes decisions were made by small groups without input from the larger team and without socializing the impact of those changes prior to. It was a guessing game as to when someone would share important or valuable news (perhaps after a bug was filed).

There were other less subtle communication costs like how little I got to know my coworkers. Imagine working together every day for six months and knowing next to nothing about them. While this might seem like a non-communication problem I’ve found the less I know someone the less context I’m able to apply to our online conversations. I’ve had a few coworkers over the years whose chat conversation style was often terse despite their very helpful demeanor in person. Knowing this helped me better know what boundaries were real and which ones were made up.

For our future (and my future self) I hope it’s possible to be in a distributed testing position and not have these barriers (or have them lessened). For Alister Scott’s organization, Automattic, it seems to be working well. In the meantime I will continue to fully enjoy working some days at home and some days in the office.

I’d like to think it’s possible to be in a distributed testing position and not have these barriers (or have them lessened).


Testers, don’t be afraid to make Production Changes

Scenario: You are testing a new page and there’s a typo. What do you do?

  • File a bug report?
  • Mention it in passing and hope the developer fixes it the next time she remembers?
  • Fix it yourself?

If you’ve got access and aren’t afraid to commit (eventual) production changes, I highly recommend making small fixes yourself. It’s a small but visible way to have an impact on the quality of your software outside of testing and writing bug reports.

A Scary Proposition

I used to be afraid of making small obvious changes. I’ve spent most of my software career testing other people’s code and when first given the chance to make and commit changes, I hesitated. It seemed so counterintuitive. What if I broke something? Who tests my change if I’m the (only) tester?

In hindsight starting small with text or label changes meant dealing with low risk bugs whose solutions are incredibly straight forward. Solution: change the spelling. That also meant my colleagues weren’t ever worried about me making things worse. Worst case is something’s still misspelled.

Learning this perspective helped me get over the initial fear and over time the benefits of me being able to fix small things meant I could reduce potential bug reports and make quick, actionable fixes that were trivial for the programmers to make. They could focus elsewhere on higher value, higher priority work while I catch and fix the small things they missed.

Today I regularly commit small changes around typos, broken links, general CSS weirdness and a few other slightly more complex things. I also add testability hooks into the production site to aid my automation code (which isn’t production bound). I enjoy this small, visible impact.

Start Small

For anyone who wants to try this I suggest starting small with the things that annoy you and that would be easier or less time consuming for you to fix, than someone else. Once you’ve made the fix have someone code review your changes (don’t worry they’ll be very easy to review initially). Over time as you learn the process and become comfortable with it, you’ll be able to tackle more difficult, yet small changes.

It’s important to set boundaries around what changes you’ll make and won’t make. I don’t jump in and mess with things I don’t understand. I also won’t volunteer to make changes being worked on by other programmers to avoid stepping on their toes.

All of this depends on your comfort level, the visibility you have into the code and your desire. If you try it and still don’t feel comfortable, or you don’t have access to your code (and don’t want to get it), then don’t worry. It’s not for everyone. For those that do, sometimes those actionable bugs are compelling enough to just fix them instead of reporting them (or in addition to). Especially if you suspect such a request is likely to be ignored for some time as low priority.

As testers it’s easy to get into this mindset of someone else writes code or fixes the bug, not me. It doesn’t have to always be that way. Sometimes it takes less effort to just fix an issue than it does to report it, wait for someone else to fix it and then test the fix. Making small fixes is really about being a team member and helping to improve quality.

Testers, don’t be afraid to make small production changes!

10 Years In

For 10 years (one whole decade) I’ve been employed in a few different software testing positions with a few different titles. It’s been a fun and challenging road. I’ve navigated large companies filled with good people and backward practices to small companies where modern practices are encouraged and my work has to stand on it’s own merit. It’s been a road of struggling to self-educate others and myself on the topic while also moving towards a more empirical model of testing.


I consider myself a reasonably experienced software tester with an interest in the human/machine balance of software development. Ever since Malcolm Gladwell popularized 1 the concept of “10,000-hours of deliberate practice for mastery of a subject”, that mark has become a subject of online discussion and a metric to peg skill levels at, even if it’s usefulness isn’t proven2. Out of curiosity, where does that leave me?

Back of the envelope calculations might go like:

  • I’ve been employed in testing for 10 years. With workdays being 8 hours and 6 of those considered productive, that’s 30 hours a week for 52 weeks a year for 10 years or 15,600 hours. Except, depending on the company, maybe only three to four hours per day was spent on testing, practicing, reporting and hopefully getting better.
    • As for hours of deliberate practice during work, I might be around the 8,000 hour mark.
  • Can’t forget to subtract out sick and vacation days.
  • Need to add in the hours outside of work where I’ve consulted, studied, taken and taught classes on testing, attended conferences and workshops and generally been active in the community
    • That should more than balance out those vacation and sick days.

I might be right at or below the 10,000-hour level of deliberate practice. Without records of my time, days and projects, it’s too difficult to get a realistic number. It’s even harder to see what the advantage or usefulness of compiling such numbers could be, except for fun (which is why I just did it).

Looking Back

When I started this journey I wasn’t sure what I wanted to do. University had come and gone without providing (or without me taking advantage of) many opportunities to learn what I liked or didn’t like. One thing I didn’t like at the time was programming. It wasn’t necessarily hard, but the value was too abstract and the examples didn’t appeal to me. After I graduated, I got a testing job by emphasizing my ability and like of documenting things (writing). Emphasizing documentation in any interview, besides technical writing, suggests I had no idea what I was doing. Looking back on those days, it seems like many of the people I was working with probably had no idea what testing was either (or if they did they decided not share or provide any feedback).

I can’t recall how many interviews I went to over the past decade where there was no testing exercise. As in no demonstration of skills necessary – just come in and try to learn about the system. We hope you do well. Maybe that’s why it took me some time to figure out if I wanted to stay in the field?

At one point I thought project management would be something I’d enjoy more than testing. It seemed more salient. After all I’d been a project leader in college on several projects and although it was stressful, it was rewarding. Eventually I got a project management certificate (nano-degree) in my spare time and came very close to taking the Project Management Professional (PMP) exam. At least until I realized the PMP, although recognizable, really didn’t mean much. In fact during the PMP training, the experienced project managers constantly reiterated there was a difference between what the PMP said to do and what you would do in real life. Many people seemed to be at that training because their employers were convinced the PMP was useful.

Happy that I had first-hand knowledge of the uselessness of the PMP (to me), I went in another direction and started learning more about software testing. I started reading books and attending conferences. The more I learned, the deeper and wider the rabbit whole became. Looking back I’m glad things turned out the way they did.

Looking Forward

It’s a little hard to think of the next decade because it’s so far away and the world is constantly changing. Here’s what I know as of this posting. I’ve been writing (blogging) about software development and testing since 2009, and writing in general since 2008 (although not consistently). That’s slowly changing and I foresee writing on a regular basis. I’ve been co-instructing BBST classes for more than a year now and I’m thinking about stepping up my role in AST and BBST.

In the near future I see more mentoring, collaborating with peers, starting a meetup and/or a workshop and hopefully producing a wider variety of public work. I’m working harder at applying programmatic solutions to problems and constantly on the look-out for examples. There’s still a lot to do.

I haven’t yet:

  • Given any talks,
  • Published any conference papers,
  • Attended any meet ups,

At 10 years in, I’m working harder than ever, on a better path and more determined. Here’s to the next decade wherever that may take me!

And nothing else funny happened

I was recently talking with someone about their testing strategy and process when I noticed they were trying to build overly-detailed test scripts (procedures). It didn’t take them long to realize specifying to such detail often left them bored (writing became redundant) and so each test became less and less detailed. I offered to take a look at their tests to see if I could help improve things and as I saw, what I consider to be, their “typical” scripted tests with each line having a step and expected result I started thinking of something Pete Walen once said:

…and nothing else funny happened.

Peter Walen, STPCon

Pete Walen dropped this gem at STPCon in San Diego a few years ago during Matt Heusser’s talk Where do Bugs Come From? Matt had just shown the Moonwalking Bear video:

I’d guesstimate at least half of the packed room hadn’t seen this. The discussion turned to scripted testing and the inherent problem of in-attentional blindness. Pete shared a past experience where he told some testers he was working with to include the phrase in the expected results of their scripted tests. It’s a kind-of-heuristic to help someone remember, even though they are focusing on one thing at depth, they need to be aware of the potentially many other things going on so that “nothing else funny happens”.

It was such a simple, powerful idea it continues to stick with me, especially when I see someone trying to specify “most everything” in their scripted tests.

As it happens, this post coincides with STPCon 2015 back in San Diego right now. There looks like a lot of awesome speakers (Matt Heusser, Michael Larsen, and Smita Mishra) but I’m particularly intrigued by Andy Tinkham’s High Volume Automation in Practice and Dave Haeffner’s Using Selenium Successfully. I wonder, is anyone live blogging?

Recognizing a problem in eBay’s iPad app

I’ve been buying and selling things on eBay for more than a decade. Naturally in the last few years I’ve spent more time on the iPhone app but for some reason I wasn’t using the iPad app. I figured it was time, so I installed the eBay app, logged into my account and went to the selling page to view my active auctions. The photo below is what I saw:

eBay iPad

Immediately I recognized a problem.

Do you see it?

On this selling page there is nothing to tell me how many bids each auction item has, assuming they have any, or if any auction will be sold. Another way to look at it: I can’t tell which auction items are going to make me money!

Confusing matters is the use of black and red colors for the current prices. Does red mean the item won’t sell? Does black mean it will? No, that doesn’t appear to be the logic. Only two of my items have bids – the third item (shown in black) and the last auction item (also shown in black). So why do the non-selling items have colors of red and black? Like I said, confusing.

Identifying problems like this can seem obvious when you have sufficient experience with a product (or someone explains it) but even without help there are ways to identify and evaluate problems such as this. All we have to do is find an oracle (a way to recognize a problem) and do some testing (perform an investigation). When reporting and evaluating problems like this I like to use the collection of consistency oracles by James Bach and Michael Bolton. You should be able to follow along fine with the rest of this essay if you’ve never seen list but it’s worth a read.

After evaluating the list of oracles I want to call attention to the ones I think help highlight the problems with the iPad app. The order below is based on my observations of the problem as I came across them. It just happens the evidence becomes stronger and more convincing as we work through the list.

  1. Inconsistent with user’s desires
  2. Inconsistent with purpose
  3. Inconsistent within product

Inconsistent with user’s desires

I think it’s reasonable to expect eBay sellers with current listings, will want to know how many of those listings have bids and if they’ve met the minimum criteria for completing the sale including reserve amounts, number of bids, etc. In fact I’d bet that’s one of the most important pieces of information they’d want to see because it’s what I wanted to see.

Wait a minute, can’t a user click on every single auction item and view more details? Yes. Assuming the user is like me and only selling a few items it’s probably a fine work-around. However, what if you’ve listed a hundred or a thousand items like eBay PowerSellers and businesses do? Do you think it’s reasonable to expect them to click on every single auction? I don’t; it defeats the purpose of the selling view.

User desires can be a hard argument to make on its own. Unless we knew eBay valued this as a high-risk area or we found it affected a large number of users, we probably need to do more research to back up our argument.

Inconsistent with purpose

I think the explicit purpose of the selling page is to help eBay sellers monitor auction items and complete sales. In fact this is what I use it for. Typically with auction durations of more than one day I will glance at each listing once per day by going to the selling page (pictured above). When the auction duration gets to be under 24 hours I will visit the page far more frequently.

Additionally by using different colors for the price of an auction that will sell vs. one that won’t, eBay can quickly identify the status of each listing item. For example if green meant an auction would sell and red meant it wouldn’t, I could easily scan through my selling page and implicitly understand how things were going. From there I could make adjustments if needed.

Since the iPad app shows the same color for auctions that will and won’t sell and because I can’t monitor my auction sales at a distance I think the sellers page is inconsistent with it’s purpose. Given my experience with the eBay product as a whole I also know it’s inconsistent with the larger eBay product.

So far we’ve covered two inconsistencies, two ways we can identify the problems I came across. In both I’m arguing, based on my experience, I understand what a user wants (as a user myself) and I understand the purpose of the product. If both of these inconsistencies sound similar that’s ok because in this case they happen to. They won’t always.

Other than my experience I don’t have much data to back up our argument. I could do more research on the product, look for claims, marketing materials, and interviews with experts that would add more evidence to our argument but let’s focus on the last inconsistency.

Inconsistent within product

As I mentioned before I’m a long time eBay user through and the iPhone app. Though and mobile apps may seem like separate products they are in fact different ways of gaining access (think distribution channel) to the same product, the eBay platform. This means we can look to those channels when thinking about product consistency.

In my mind, even though the iPad app fails to fulfill its users desires and purpose it is most obviously inconsistent with other aspects of the eBay product. Here’s the same seller’s view on the eBay iPhone app:

eBay iPhone seller view 1

eBay iPhone seller view 2

Observe any differences?

The iPhone version shows a similar layout but with a few important differences:

  1. The iPhone app lists the number of bids on each listing right away. I don’t have to go to any other detailed view to get this information (also re-affirms it’s importance).
  2. The current price is color coded in a way that fits with our normal assumptions – green means a listing is going to sell and red means it won’t. This makes it far easier for us to monitor our current auctions at a glance and it fulfills it’s purpose better than the iPad app.

If we were to compare either mobile app to, we’d see the sellers page is only consistent with the iPhone app. This is the strongest evidence we’ve found that points to problems with the iPad app. It’s a far more credible argument than the others because it seems likely eBay would want it’s iPad channel consistent with its others ( and iPhone app). Without doing much additional research we’ve found a way to explain to eBay stakeholders that these problems exist and are bad.

Closing Thoughts

Usually I report problems so they get fixed. Yet identifying, evaluating and then describing a problem in such a way it convinces someone else of its importance isn’t as simple as you might initially think. If I did this well I’ve convinced you, the reader, there’s a problem with the selling page of the iPad app.

Now all I have to do is file the bug with eBay and delete my eBay iPad app. Why delete the app? Well I’ve been trying to sell things on eBay for the past month and given its problems there’s very little value in me using it.