Shaping Your Identity as a Tester

On Thursday, June 25, 2015 I presented my first webinar called Shaping Your Identity as a Tester that was based on an earlier article I wrote called Blogging for your Career.

uTest recorded it and made it part of their uTest University series, you can check it out here. I’ve also embedded the video below for easy viewing. The slides are pretty simple. My part lasts less than 30 minutes, the rest of the time is spent answering questions fielded from the 75 or so participants who joined. Check it out:

If you joined the webinar and your question wasn’t answered or if you have any questions after watching it here, feel free to leave me a comment (or contact information) or connect with me (on the right) and I’d be happy to respond!

The Apple Watch won’t change Testing

Probably. The Apple Watch won’t change Testing, probably.

Last month uTest announced a contest to win an Apple Watch. All you had to do was provide a response to this post:

In just a paragraph, describe how the Apple Watch will or will not change software testing as we know it today, or how testers are testing.

42mm Apple Watch

While I probably should have clarified the rules a bit (how do you define a paragraph?), I responded with:

If software testing is an empirical, technical investigation of a product for the stakeholders and a new product is introduced, that doesn’t change what software testing is or why we test. It might add some new dimensions to the product (the watch being an extension of the phone, tactile touch interface, etc.) that as testers we have to consider. It might change the importance or risk of some of those dimensions (change in interface, platform, etc.). Or it might change which test techniques we apply and how we apply them (think of stress testing a watch or computer aided testing / automation) but it probably won’t change testing.

I feel like elaborating. (tl;dr skip to the last paragraph)

As I was trying to formulate an answer I was thinking about how a test strategy might change between two similar or complimentary devices – an iPhone and an Apple Watch. The differences might suggest what changes were necessary to the model I was using. That model looked something like the Heuristic Test Strategy Model and the changes I noticed were within product elements and test techniques.

For example we might see a difference between the iPhone and the Apple Watch in:

  • Operations. I imagine the environment and general use cases are a bit different and more extreme for a watch. I’m extremely careful about damaging my phone but I seem to always strike my arms and wrist against doors and walls without knowing it. The fitbit I wear around my non-dominant wrist speaks to this extreme or disfavored use.
  • Platforms. The hardware and software are different. The OS that runs on the watch is new (watchOS), in addition to the apps. What level of support does the iPhone provide (its a required external companion) to the Apple Watch?
  • Interfaces. The user interface seems like the most obvious different given the small display and crown as the home screen. What about the new charging interface or how data is exchanged between the watch and the phone?

Those are just a few dimensions of the Apple Watch I could think of in the thirty or so minutes I took. How many more am I forgetting? (We should examine as many of those dimensions as possible when testing).

Then I started looking at the top Test Techniques listed in the HTSM. How many of them can we apply to testing mobile devices like an iPhone and now the Apple Watch?

  • Function Testing
  • Domain Testing
  • Stress Testing
  • Flow Testing
  • Scenario Testing
  • Claims Testing
  • User Testing
  • Risk Testing
  • Automated Testing

All of them! The challenge might be in applying these techniques. I’ve heard mobile GUI automation like Appium has come a long way in a short time but still has problems and isn’t at the level of Selenium WebDriver. My own experience with mobile simulators suggests they are much less reliable than their desktop counterparts.

After going through this brief exercise I came away thinking this model was still just as applicable. Although the business case is still being made amongst consumers and without knowing any specific project factors, my testing thought process remained much the same. This isn’t to say there isn’t a need for a specialty of testers who are really good at applying test techniques to mobile devices; only that the mental model is still the same. Testing as a whole isn’t really changing.

Why It Matters

I’ve always found joy in exploring new products and the impact they have on our lives. Although it’s assumed when you work in technology you are a technophile – someone who loves technology and you live and breathe gadgets and software -that’s not always the case. I find it just as interesting how quickly I abandoned certain things as how much they stick to me and how much I use them.

I’m still evaluating the Apple Watch. As progress slows on the development of smartphones I’m starting to question the relentless upgrade process – waiting instead for things I feel are worthy of spending the money on. The Apple Watch falls into this same category. I don’t typically wear watches but as I said before I do wear a fitbit. I like knowing how active or inactive I am. Whether I’m ready for a relentless upgrade cycle of expensive watches over the next 5 years in addition to whatever new phones comes out is an entirely different story.

As for the uTest May contest, I came in third place. Thanks to uTest and everyone who voted! Maybe if I win a few more contests I’ll be able to justify getting my own Apple Watch? Even though it probably won’t change testing much, how can I say no?

When to use a Gemfile

I’ve been building a GUI acceptance test automation suite locally in Ruby using the RSpec framework.  When it was time to get the tests running remotely on Sauce Labs, I ran into the following error:

RSpec::Core::ExampleGroup::WrongScopeError: `example` is not available from within an example (e.g. an `it` block) or from constructs that run in the scope of an example (e.g. `before`, `let`, etc). It is only available on an example group (e.g. a `describe` or `context` block).
occurred at /usr/local/rvm/gems/ruby-2.1.2/gems/rspec-core-3.2.2/lib/rspec/core/example_group.rb:642:in `method_missing'

It took a few minutes debugging before I spotted the error:

../gems/ruby-2.1.2/gems/rspec-core-3.2.2/lib/rspec/core/..

Source of problem: My remote tests were using a different version of RSpec than I was locally. Solution: Create a Gemfile to specify the version of using Rspec I’m using.

Since I didn’t realize I needed a Gemfile my question was, in general, when should someone use a Gemfile? According to the manual, a Gemfile

describes the gem dependencies required to execute associated Ruby code. Place the Gemfile in the root of the directory containing the associated code.

For example, in this context, I would place a Gemfile into any folder where I specifically want to call tests to run. In my case that meant a few specific locations:

  • At the root of the folder – where I run the whole suite of tests
  • In the /spec/ folder – where I typically run tests at an individual level

At a minimum I’d specify:

  • A Global Source
  • Each Gem I use locally that Sauce Labs will need to use

In the end it might look something like this:

Test, adapt, and re-test.

See you at CAST 2015

Thanks to my awesome company’s sponsorship and my wife’s love of travel I will be at CAST this year and attending the tutorial “Delivering Difficult Messages” by Fiona Charles. This is both my first time attending CAST and traveling to Grand Rapids, MI so I’m expecting to have a little fun and to learn a lot. As for the conference experience, I’m not sure what to expect.

Over the last four or five years I’ve been to a few different conferences and training events:

WTST 13 was my first-ever peer workshop and the last trip I made. In contrast to large conferences like STAR and STPCon where several hundred or even a thousand people crisscross each other as they go to any number of tutorials or talks, the peer workshop format was limited to 20-25 participants in a single room. There were a few presenters who spoke along a similar theme but it was the participants who drove the discussions until everyone was satisfied. It was certainly a unique experience.

Comparing workshops to conferences may not be fair due to their different approaches but my goal for attending them is the same: to learn something new and useful (apply to my job or company) and interact with my peers.

My understanding of CAST’s format, even before going, is it’s a small-ish conference attended by few hundred conference-goers, most of whom are AST members. Although a few hundred participants is still a good-sized event, I’m hoping to be able to interact with a few (or a lot) people. Of the few conferences I’ve attended the biggest impact has always been the interactions: meeting others, sharing problems, being challenged in my thinking, trying to explain something, reference recommendations, etc.

Aside from the one tutorial I’m signed up for and a welcome get-together in the morning each day I don’t know what else there is planned. There’s no schedule available, yet and until there is I won’t be sure what to expect.

Enough about my expectations; who else is attending and what are you looking forward to?

And nothing else funny happened

I was recently talking with someone about their strategy and overall testing process when I noticed they were trying to build overly-detailed test scripts. It didn’t take them long to realize specifying to such detail often left them bored (writing became redundant) and so each test became less and less detailed. I offered to take a look at their tests to see if I could help improve things and as I saw, what I consider to be, their “typical” scripted tests with each line having a step and expected result I started thinking of something Pete Walen once said:

…and nothing else funny happened.

Pete Walen dropped this line at STPCon in San Diego a few years ago during Matt Heusser’s talk Where do Bugs Come From? Matt had just shown the Moonwalking Bear video:

I’d guesstimate at least half of the packed room hadn’t seen this. The discussion turned to scripted testing and the inherent problem of in-attentional blindness. Pete shared a past experience where he told some testers he was working with to include the phrase in the expected results of their scripted tests. It’s a kind-of-heuristic to help someone remember, even though they are focusing on one thing at depth, they need to be aware of the potentially many other things going on so that “nothing else funny happens”.

It was such a simple, powerful idea it continues to stick with me, especially when I see someone trying to specify “most everything” in their scripted tests.
(more…)

Recognizing a problem in eBay’s iPad app

I’ve been buying and selling things on eBay for more than a decade. Naturally in the last few years I’ve spent more time on the iPhone app but for some reason I wasn’t using the iPad app. I figured it was time, so I installed the eBay app, logged into my account and went to the selling page to view my active auctions. The photo below is what I saw:

eBay iPad

Immediately I recognized a problem.

Do you see it?

On this selling page there is nothing to tell me how many bids each auction item has, assuming they have any, or if any auction will be sold. Another way to look at it: I can’t tell which auction items are going to make me money!

Confusing matters is the use of black and red colors for the current prices. Does red mean the item won’t sell? Does black mean it will? No, that doesn’t appear to be the logic. Only two of my items have bids – the third item (shown in black) and the last auction item (also shown in black). So why do the non-selling items have colors of red and black? Like I said, confusing.

Identifying problems like this can seem obvious when you have sufficient experience with a product (or someone explains it) but even without help there are ways to identify and evaluate problems such as this. All we have to do is find an oracle (a way to recognize a problem) and do some testing (perform an investigation). When reporting and evaluating problems like this I like to use the collection of consistency oracles by James Bach and Michael Bolton. You should be able to follow along fine with the rest of this essay if you’ve never seen list but it’s worth a read.

After evaluating the list of oracles I want to call attention to the ones I think help highlight the problems with the iPad app. The order below is based on my observations of the problem as I came across them. It just happens the evidence becomes stronger and more convincing as we work through the list.

  1. Inconsistent with user’s desires
  2. Inconsistent with purpose
  3. Inconsistent within product

Inconsistent with user’s desires

I think it’s reasonable to expect eBay sellers with current listings, will want to know how many of those listings have bids and if they’ve met the minimum criteria for completing the sale including reserve amounts, number of bids, etc. In fact I’d bet that’s one of the most important pieces of information they’d want to see because it’s what I wanted to see.

Wait a minute, can’t a user click on every single auction item and view more details? Yes. Assuming the user is like me and only selling a few items it’s probably a fine work-around. However, what if you’ve listed a hundred or a thousand items like eBay PowerSellers and businesses do? Do you think it’s reasonable to expect them to click on every single auction? I don’t; it defeats the purpose of the selling view.

User desires can be a hard argument to make on its own. Unless we knew eBay valued this as a high-risk area or we found it affected a large number of users, we probably need to do more research to back up our argument.

Inconsistent with purpose

I think the explicit purpose of the selling page is to help eBay sellers monitor auction items and complete sales. In fact this is what I use it for. Typically with auction durations of more than one day I will glance at each listing once per day by going to the selling page (pictured above). When the auction duration gets to be under 24 hours I will visit the page far more frequently.

Additionally by using different colors for the price of an auction that will sell vs. one that won’t, eBay can quickly identify the status of each listing item. For example if green meant an auction would sell and red meant it wouldn’t, I could easily scan through my selling page and implicitly understand how things were going. From there I could make adjustments if needed.

Since the iPad app shows the same color for auctions that will and won’t sell and because I can’t monitor my auction sales at a distance I think the sellers page is inconsistent with it’s purpose. Given my experience with the eBay product as a whole I also know it’s inconsistent with the larger eBay product.

So far we’ve covered two inconsistencies, two ways we can identify the problems I came across. In both I’m arguing, based on my experience, I understand what a user wants (as a user myself) and I understand the purpose of the product. If both of these inconsistencies sound similar that’s ok because in this case they happen to. They won’t always.

Other than my experience I don’t have much data to back up our argument. I could do more research on the product, look for claims, marketing materials, and interviews with experts that would add more evidence to our argument but let’s focus on the last inconsistency.

Inconsistent within product

As I mentioned before I’m a long time eBay user through eBay.com and the iPhone app. Though eBay.com and mobile apps may seem like separate products they are in fact different ways of gaining access (think distribution channel) to the same product, the eBay platform. This means we can look to those channels when thinking about product consistency.

In my mind, even though the iPad app fails to fulfill its users desires and purpose it is most obviously inconsistent with other aspects of the eBay product. Here’s the same seller’s view on the eBay iPhone app:

eBay iPhone seller view 1

eBay iPhone seller view 2

Observe any differences?

The iPhone version shows a similar layout but with a few important differences:

  1. The iPhone app lists the number of bids on each listing right away. I don’t have to go to any other detailed view to get this information (also re-affirms it’s importance).
  2. The current price is color coded in a way that fits with our normal assumptions – green means a listing is going to sell and red means it won’t. This makes it far easier for us to monitor our current auctions at a glance and it fulfills it’s purpose better than the iPad app.

If we were to compare either mobile app to eBay.com, we’d see the sellers page is only consistent with the iPhone app. This is the strongest evidence we’ve found that points to problems with the iPad app. It’s a far more credible argument than the others because it seems likely eBay would want it’s iPad channel consistent with its others (eBay.com and iPhone app). Without doing much additional research we’ve found a way to explain to eBay stakeholders that these problems exist and are bad.

Closing Thoughts

Usually I report problems so they get fixed. Yet identifying, evaluating and then describing a problem in such a way it convinces someone else of its importance isn’t as simple as you might initially think. If I did this well I’ve convinced you, the reader, there’s a problem with the selling page of the iPad app.

Now all I have to do is file the bug with eBay and delete my eBay iPad app. Why delete the app? Well I’ve been trying to sell things on eBay for the past month and given its problems there’s very little value in me using it.

References

State of Testing Survey 2015

In December of 2013 I mentioned Lalit Bhamare and Joel Montvelisky created a survey for assessing the “state” of the testing community to help the community to get a better understanding of what is going on around the world and help testers improve things. They published their results in a report that’s worth a read (PDF).

It’s 2015 and since I think the survey is a worthwhile effort I’ll do my “civic duty” and participate in this years survey. I encourage you to do the same by going here.

Here’s your official save-the-date:

https://twitter.com/PractiTest/status/557076285386932224

Please sign the Petition to Stop ISO 29119

I signed the Petition to Stop ISO 29119 and I think so should you too. Here’s how: sign this petition!

I don’t typically get involved in overly political movements, typically because I don’t feel I have enough information to make an educated or defensible position. I think software engineering standards (including ISO 29119) are one of those overly political movements that aren’t designed to benefit the majority of the community nor do they have practitioners in mind (part of the challenge in trying to create a standard I’m sure).

There are a few reasons why I’m worried about this standard:

  • Closed standards seem to be anti-thesis of the modern software age where open-source standards, software and collaboration are the keys to success. Plus it makes the organization seem a bit shady.
  • It’s anti-agility. Burdensome test processes, documentation, etc. over the skills and experience of the people applying them directly reject the philosophy of the Agile Manifesto.
  • There are legitimate disagreements within the software testing community on fundamental things such as terminology, let alone basic processes. In the last 15+ years as an industry we’ve started to develop a path towards understanding these disagreements but I highly doubt this “standard” has solved or even attempted to understand and settle those differences.

James Bach says “[a] standard for testing would have to reflect the values and practices of the world community of testers. Yet, the concerns of the Context-Driven School of thought, which has been in development for at least 15 years have been ignored and our values shredded by this so-called standard and the process used to create it. They have done this by excluding us.” (ref 1) (emphasis is my own)

  • As Cem Kaner said “Standards are political documents and sometimes legal ones. The existence of a standard makes it easier for a court (or a regulator) to rule that the standard-approved approach is the professionally correct one, and the non-approved approaches (or the ones that conflict with the approved one) are professionally incorrect and therefore improper. The imposition of a standard that imposes practices and views on a community that would not otherwise agree to them, is a political power play.” (ref 2) (emphasis is my own)

In the end I’m not sure how much effect signing this petition will have on it’s outcome, impact or acceptance but I am concerned enough to make a stance. I would rather say I protested it than accept it in silence.

I signed the Petition to Stop ISO 29119 and I think so should you too.  Go to iPetitions to sign.

References:

  1. James Bach’s post
  2. Cem Kaner’s post via Context Driven Testing

Getting better at Domain Testing

In late January of 2014, after the Workshop on Teaching Software Testing (WTST) at Florida Institute of Technology, Dr. Cem Kaner and Dr. Rebecca Fiedler put together a 5-day pilot course to beta test a new Black Box Software Testing (BBST) course called Domain Testing. I was one of ten participants to try it out.

Then they announced a follow up / pilot class for BBST Domain Testing that would be online. I applied to be part of the class and was accepted. This was like an expanded version of the pilot course (less intense than 5 full days) but more similar to what you might expect from a BBST courses. The course focused from Day One on diving deeper into the test technique Domain Testing and working through various aspects of the Domain Testing schema until we got to a final / capstone project where we put everything together.

I’m happy to share that I’ve officially completed the course!

TDD and Software Testers

I’ve been following along with the series of conversations with Martin Fowler, Kent Beck and David Heinemeier Hansson (DHH) entitled Is TDD Dead. The whole conversation about what’s good, bad and ugly with test driven development (TDD) is interesting in my role as a software tester and from an overall system / quality perspective. What works, what doesn’t? What do some programmers like about it and what do others fear? Does TDD translate into a better product? Etc.

According to Fowler’s website, part 3 of the series covers

…the various ways in which we get feedback while programming and the role of QA in providing feedback to developers.

The whole series is worth a watch but if you are just interested in TDD and the role it plays when you have software testers (or QA), watch it here:

The three people involved with it have have varying experiences with Fowler having worked for many years with software testers in enterprise software, Beck now working at Facebook where they have few testers (and his own experience with dysfunctional QA) and DHH’s experience running Basecamp. It’s an interesting and relevant discussion because it’s coming from a programmers point of view (programmer testing).  My view says testing is an investigation designed to reveal information about a product. Beck frames it as feedback that builds confidence in the code. I think both views of the software are valuable and those differences in techniques and approaches yield very different ways of viewing quality.

The title “TDD is dead” reminds me of the saying “Test is dead”. Neither of those titles are accurate (they are catchy) but understanding the differences in views can help us when talking to stakeholders who have similar feelings or views.