Getting Paid to Learn

Picking up experience on the job can be great (for example learning about SEO, application monitoring or observability) but all things being equal, it shouldn’t come at the expense of the many other ways we learn. Without the support to continuously learn new things we get stuck in jobs and become less valuable to the market and our employers over time. That’s why I’m a big fan of the concept of Getting Paid to Learn.

Getting Paid to Learn

Just like it sounds, I’m talking about companies that help and support their employees learn and develop new skills over time by giving them time and budget to learn new things.

Getting the Time

I joined BloomNation over a year ago and began learning how to write code in a brand new language: JavaScript. My boss knew this meant giving me time to learn and time to produce something at a far slower rate than I might normally be able to. (Note: this is time during the regular work day) The upside of this meant alignment with my front-end developers and now I have some skill that I can use again and again in other areas like in helping to do front-end Code Reviews.

Getting the Budget

Years ago when I first started learning how to write UI automation I requested my then company buy me an online book. As I used my new skill to tackle newer and larger problems I outgrew the advice a book could give me and asked for the budget to hire a known consultant to help guide me in more advanced practices. We set a budget and hired that consultant part time.

While you don’t necessarily have to hire a consultant, having a defined (or loosely defined) annual budget per person for gaining education through online courses, books (FYI anyone can gift you a Kindle eBook fairly easily), conferences, is a huge signal that the company values you, is invested in your growth and its also a pretty nice recruitment tool.

Peace of Mind

My own experience tells me companies are starting to get the hint about having an active policy toward continuous learning. Some have enterprise training accounts with Udemy, Udacity, Safari Online or other programs. Others have a defined budget per year. This has become one of my goto questions when I evaluate a company to join: do they offer both budget and time to learn new things?

While I may not always have the time to take advantage, when I do want to learn something new, upskill or generally increase my depth of knowledge I want the safety and confidence of knowing the company is willing to pay me to learn.

Do things that don’t scale

If you can find someone with a problem that needs solving and you can solve it manually, go ahead and do that for as long as you can, and then gradually automate the bottlenecks. – Paul Graham

With the title of Automation Engineer it might seem like automating bottlenecks is “my jam” and therefore “do[ing] things that don’t scale” might seem kind of an odd thing for me to agree with. However it turns out this is a useful way to think through solutions to problems, including testing problems.

From a business standpoint this makes a lot of sense, why waste time automating something until you are sure it has value? Many startups have blown through money building solutions to things (or automating problems) that aren’t actually important (or real problems). If you are an established company and thinking of branching into a new market segment, it’s ok to do everything by hand at first: signing up customers, placing orders, building website, etc.

From a software standpoint this also makes a lot of sense. Don’t worry about automating bottlenecks like test setup or execution until:

  1. You are sure the tests have value (business or technical) AND
  2. They’ve become bottlenecks to delivery

Otherwise don’t waste your time.

How to Write a Good Bug Report, use RIMGEN

RIMGEN is an acronym and mnemonic (or memory aid) to help us remember the elements of a good bug report. It can be used to help anyone write a better bug report or review existing bug reports looking to make improvements. 

In general my preference with reporting bugs is to:

  1. Work with a developer to get them fixed, either by mentioning it passing or pairing with them. By far the most effective way to get bugs fixed.
  2. Fix it myself, although most of these are very simple bugs.
  3. Write a bug report. These are good instructions for that process.

How to Write a Good Bug Report

Writing a bug report seems simple on the surface. People in many roles (programmers, testers, internal and external customers, analysts) write them without much, if any, practice. Although we’ve all seen those bug reports come in without much detail or context and guess what? They don’t get fixed. Just because something is easy to do doesn’t mean it is easy to do well.

We’re writing a good bug report to get a problem fixed. Programmers are busy people, with lots of things going on and plenty of demands on their time. They don’t want to look at a bug report that doesn’t make sense, requires too much work on their part to understand or is offensive. Let’s be honest, if a bug is fairly obvious a programmer isn’t likely to need much information, motivation or influence to fix it. They might not even require a bug report to be written (mention it in passing or through slack). Those aren’t the bugs we are interested in advocating for. We’re advocating for the bugs or problems we understand to be important (for some reason) but might not be easily understood as such.

How do we advocate for bugs to be fixed? We develop reports that clearly communicate the problem, reflect the problem in a harsh but honest way so that some stakeholder realizes the risk and wants to them fixed. Just reproducing the problem might not always be enough. We want to make programmers want to fix the bug by influencing their decision, we’re selling them on fixing the bugs. To “sell” someone on fixing the bugs we need to write a good report.

To help us understand the elements of a good report we can use the heuristic RIMGEN.

RIMGEN:

R – Replicate it

When writing a bug report we should include enough information to get the bug fixed including the steps or conditions necessary to replicate the problem on another machine. The lecture recommends using at least 2 different machines – a powerhouse computer and another less powerful / resource constrained machine. (Virtual environments might work as well.)

I – Isolate it

After we’ve replicated the problem, we want to identify the shortest number of critical steps necessary to replicate the problem by eliminating some of the variables. Clarity is the goal.

We also want to make sure to write up only one problem per report so we don’t get into a situation where one part of the bug is fixed and the other remains open. Example: OpenOffice Impress crashes after importing a file, but our reproduction is 25 steps long. We do some more testing that helps us eliminate some variables until we’re able to get the problem replicated in 6 steps.

M – Maximize it

Maximizing the problem means doing some follow up testing to see if we can find a more serious problem than the one we were originally going to report. Follow up testing can consist of varying our behavior, our environment, our inputs or even our system configuration. The more serious the problem the more likely (or easier) it will be to convince someone to fix it. Example: We find a problem in OO Impress where it won’t import old PowerPoint files; the application just won’t respond. After some follow up testing we find Impress actually crashes if we use an old PowerPoint file over a certain size.

Maximizing a bug (or follow up activities as it’s also known) is one of the more interesting and time consuming aspects of writing a bug report. There’s quite a bit of detail just in the four categories of variables: vary our behavior, vary our environment, vary our data and vary our system configuration and for those interested Markus Gartner goes a little more in-depth about each in this post.

G – Generalize it

Generalizing the bug means seeing if the problem affects more people in more ways than we originally thought. If the bug was a corner case maybe we can un-corner it, show that it occurs under less extreme conditions. (replicating on a broader range of systems?) Example: Same problem as above in OO Impress where it crashes when we try to import old PowerPoint files over a certain size. We vary the data (input) and find out Impress actually crashes when we try to import any file type over a certain size. Now the problem looks really bad and affects more users than just older PowerPoint users.

E – Externalize it

Switch the focus of the report from the program failing to the stakeholders who will be affected by it and show how it will affect them. (Historical review?) Example: OO Impress crashing when importing any file type of a certain size will stop all users from importing comparable application files (like from PowerPoint). This could prevent new users from downloading and installing the application and even turn off existing users who rely on the compatibility.

N – Neutral Tone

Make sure the bug report is easy to understand and that the tone of the report is neutral so as not to offend anyone reading it (perhaps the developer who created the bug) and/or discredit your reputation as the bug reporter (at least in the eye of the reader).

BUG TITLES AND SUMMARIES

To help us communicate (and “sell”) the problem clearly in bug titles and summaries we can use the “Fails, When Protocol”. “Fails, When” says describe the failure first and then when that failure happens second.

For example in Bug Advocacy we look at two (apparently) well known problems in Microsoft Paint 95 (I think that’s the version) related to cutting and pasting from a zoomed section. When I wrote up my problem summary (before understanding the protocol) it came out as: “Cutting (Ctrl-X) a selected area while zoomed causes inconsistent behaviors.” The problem with this summary was it was too vague, contained minor details about the problem and although I spotted two bugs I tried to combine the two summaries into one. If I had used the “Fails, When” protocol a more appropriate set of bug titles might have been: “Cut doesn’t cut (freehand, zoom)” or “Cuts the wrong area (freehand, zoom, resize window)”.

To help us avoid being vague when summarizing and communicating a problem it’s important to consider how much insight the title or summary gives you into the nature, severity, or scope of the problem; does it over emphasize minor details (especially for the bug title)?

A Few Other Things

  • Looking for something a little more portable? Checkout my Bug Reporting Guidelines document.
  • This is an update of my previous post “How to Write a Good Bug Report (and be a Bug Advocate)” with some slight alterations.
  • Design vs coding bugs are slightly different and have a different emphasis using RIMGEN. I hope to address this later in a separate article.
  • Advocating for a bug to be fixed is not an invitation to be sleazy or aggressive in your reporting. We are trying to influence a decision based on our research and understanding, not trying to ruin our credibility by pushing something the business has no desire to fix.
  • RIMGEN is the updated version of RIMGEA

Recognizing a problem in eBay’s iPad app

I’ve been buying and selling things on eBay for more than a decade. Naturally in the last few years I’ve spent more time on the iPhone app but for some reason I wasn’t using the iPad app. I figured it was time, so I installed the eBay app, logged into my account and went to the selling page to view my active auctions. The photo below is what I saw:

eBay iPad

Immediately I recognized a problem.

Do you see it?

On this selling page there is nothing to tell me how many bids each auction item has, assuming they have any, or if any auction will be sold. Another way to look at it: I can’t tell which auction items are going to make me money!

Confusing matters is the use of black and red colors for the current prices. Does red mean the item won’t sell? Does black mean it will? No, that doesn’t appear to be the logic. Only two of my items have bids – the third item (shown in black) and the last auction item (also shown in black). So why do the non-selling items have colors of red and black? Like I said, confusing.

Identifying problems like this can seem obvious when you have sufficient experience with a product (or someone explains it) but even without help there are ways to identify and evaluate problems such as this. All we have to do is find an oracle (a way to recognize a problem) and do some testing (perform an investigation). When reporting and evaluating problems like this I like to use the collection of consistency oracles by James Bach and Michael Bolton. You should be able to follow along fine with the rest of this essay if you’ve never seen list but it’s worth a read.

After evaluating the list of oracles I want to call attention to the ones I think help highlight the problems with the iPad app. The order below is based on my observations of the problem as I came across them. It just happens the evidence becomes stronger and more convincing as we work through the list.

  1. Inconsistent with user’s desires
  2. Inconsistent with purpose
  3. Inconsistent within product

Inconsistent with user’s desires

I think it’s reasonable to expect eBay sellers with current listings, will want to know how many of those listings have bids and if they’ve met the minimum criteria for completing the sale including reserve amounts, number of bids, etc. In fact I’d bet that’s one of the most important pieces of information they’d want to see because it’s what I wanted to see.

Wait a minute, can’t a user click on every single auction item and view more details? Yes. Assuming the user is like me and only selling a few items it’s probably a fine work-around. However, what if you’ve listed a hundred or a thousand items like eBay PowerSellers and businesses do? Do you think it’s reasonable to expect them to click on every single auction? I don’t; it defeats the purpose of the selling view.

User desires can be a hard argument to make on its own. Unless we knew eBay valued this as a high-risk area or we found it affected a large number of users, we probably need to do more research to back up our argument.

Inconsistent with purpose

I think the explicit purpose of the selling page is to help eBay sellers monitor auction items and complete sales. In fact this is what I use it for. Typically with auction durations of more than one day I will glance at each listing once per day by going to the selling page (pictured above). When the auction duration gets to be under 24 hours I will visit the page far more frequently.

Additionally by using different colors for the price of an auction that will sell vs. one that won’t, eBay can quickly identify the status of each listing item. For example if green meant an auction would sell and red meant it wouldn’t, I could easily scan through my selling page and implicitly understand how things were going. From there I could make adjustments if needed.

Since the iPad app shows the same color for auctions that will and won’t sell and because I can’t monitor my auction sales at a distance I think the sellers page is inconsistent with it’s purpose. Given my experience with the eBay product as a whole I also know it’s inconsistent with the larger eBay product.

So far we’ve covered two inconsistencies, two ways we can identify the problems I came across. In both I’m arguing, based on my experience, I understand what a user wants (as a user myself) and I understand the purpose of the product. If both of these inconsistencies sound similar that’s ok because in this case they happen to. They won’t always.

Other than my experience I don’t have much data to back up our argument. I could do more research on the product, look for claims, marketing materials, and interviews with experts that would add more evidence to our argument but let’s focus on the last inconsistency.

Inconsistent within product

As I mentioned before I’m a long time eBay user through eBay.com and the iPhone app. Though eBay.com and mobile apps may seem like separate products they are in fact different ways of gaining access (think distribution channel) to the same product, the eBay platform. This means we can look to those channels when thinking about product consistency.

In my mind, even though the iPad app fails to fulfill its users desires and purpose it is most obviously inconsistent with other aspects of the eBay product. Here’s the same seller’s view on the eBay iPhone app:

eBay iPhone seller view 1

eBay iPhone seller view 2

Observe any differences?

The iPhone version shows a similar layout but with a few important differences:

  1. The iPhone app lists the number of bids on each listing right away. I don’t have to go to any other detailed view to get this information (also re-affirms it’s importance).
  2. The current price is color coded in a way that fits with our normal assumptions – green means a listing is going to sell and red means it won’t. This makes it far easier for us to monitor our current auctions at a glance and it fulfills it’s purpose better than the iPad app.

If we were to compare either mobile app to eBay.com, we’d see the sellers page is only consistent with the iPhone app. This is the strongest evidence we’ve found that points to problems with the iPad app. It’s a far more credible argument than the others because it seems likely eBay would want it’s iPad channel consistent with its others (eBay.com and iPhone app). Without doing much additional research we’ve found a way to explain to eBay stakeholders that these problems exist and are bad.

Closing Thoughts

Usually I report problems so they get fixed. Yet identifying, evaluating and then describing a problem in such a way it convinces someone else of its importance isn’t as simple as you might initially think. If I did this well I’ve convinced you, the reader, there’s a problem with the selling page of the iPad app.

Now all I have to do is file the bug with eBay and delete my eBay iPad app. Why delete the app? Well I’ve been trying to sell things on eBay for the past month and given its problems there’s very little value in me using it.

References

First principle reasoning

When I was young I remember wanting to be an awesome football player like Joe Montana, or an FBI agent working on the X-Files like Fox Mulder. These days I want to have the skills to identify and solve problems like Elon Musk.Musk is an interesting person. He’s created and sold numerous companies and with the profits he’s created a rocket building / space exploration company that is now the first (private) company to make it to space. He’s also built an American electric car company. While all these things make Musk an interesting person on the surface, its his approach that makes him enviable.In his TED talk Musk credits his training in physics with his ability to see and understand difficult problems. He says physics is about how to discover new things that might seem counterintuitive and physics first principle provides a good framework for thinking. The video is a good conversation between Elon Musk and TED curator Chris Anderson, I recommend watching it. Musk mentions first principle reasoning at about 19:37:

According to Wikipedia first principle in physics means you start directly at the lowest levels – at the laws. Musk provides a slightly easier understanding saying first principle is reasoning from the ground up as opposed to reasoning by analogy, which is copying what other people do with slight variations.

Musk further elaborates on first principle reasoning in this video. To summarize he says you look at the fundamentals of things, make sense of it, construct your reasoning and conclusion and (if possible) compare that to whatever is the current understanding. A part of that process involves questioning conclusions, asking whether or not something could be true. Sounds like Musk’s constantly modeling, learning, testing and re-modeling.

In thinking about my job there seems to be more reasoning by analogy than perhaps there should be (or at least its obvious to someone new). Whenever one of my developers or I ask why something is, why some conclusion has been reached – the typical response is “that’s how its done”. If I ask my test team why they do something a certain way its always “that’s how we’ve always done it” and there seems to be no desire (at least I haven’t seen it yet) to know whether something makes sense or is based on a real understanding of the problem. Perhaps there should be more modeling, learning and testing?

We all do some reasoning by analogy, in many ways it’s a much simpler way to communicate and learn but for many of us in the software engineering fields (testers and developers) perhaps we confuse reasoning methods? So how do we determine when we need to use first principles and when its ok to use analogy in reasoning? That’s the million-dollar question. I think we do like Musk: create a model, ask questions to help us learn, test and when we aren’t satisfied with the answer, we reason from the ground up.

Low and High Intensity Learning

Paul Graham in his essay Wealth says startups are a way of compressing a whole working life into a few years. You work at a very high intensity for a short period of time (say four years) instead of the normal low intensity for a long period of time (say forty years), in other words startups are a way of increasing your productivity exponentially. In my experience there is a correlation between high intensity working and high intensity learning.

The potential relationships between high-intensity work and learning have a lot of appeal because it provides a chance to leapfrog our understanding of several domains in a short period of time.

In his essay Startup = Growth Graham defines a startup as “… a company designed to grow fast”. My last company was small, we considered ourselves a startup (although according to Graham’s definition we were not) but we worked considerably faster (higher intensity) than a larger company would have. I can say that with some certainty now because a 15,000-person company acquired our small (maybe 10-person) company and the differences are pretty dramatic.

To be fair there is a difference between the learning that occurs when a person is working for a high intensity company as opposed to doing their own high intensity work. With a high intensity company people learn whatever they have to in order to solve the problems in front of them, then they move on. A person working intensely on their own has the freedom to focus on what they want but they run the risk of never finding focus.

I’m struggling with the second part. I’m back to a low-intensity company but I don’t want to be pulled into a low-intensity learning situation. Part of me says that won’t happen because of my own internal drive but another part of me is worried the low-intensity rhythm of the company will make it hard to find focus. (I’m not saying my small company was a good example of a high intensity work / learning environment but as of right now it seems better than the corporate world.)

One solution is to create my own startup – something designed to grow fast which would force fast learning. It would be amazing for many reasons including getting back to a higher-intensity learning situation but I don’t know where to start. Another solution is to find a person or person(s) with the same interests or goals as I have and work together to learn. Maybe the small team size would have the effect of pushing each other into a higher intensity work and learning environment? Now where do I begin?