Exploratory Testing FAQs

I’ve come across a number of Frequently Asked Questions about Exploratory Testing and I’ve got what I hope are pretty good answers.

Exploratory Testing FAQs

Frequently Asked Questions about exploratory testing. Got a quick question? Get a quick answer.

Yes, there are many examples where people have used tools to enable and enhance their exploratory testing.

Is all testing exploratory?

No. Not unless you change the definition of testing to specifically exclude testing done by machines.

What is Exploratory Testing?

ET is an approach (or style) to testing that emphasizes the individual tester focusing on the value of their work through continuous learning and design.

Is Exploratory Testing used in Agile teams?

Yes. ET is about optimizing the value of your work given your context and so it’s a natural fit in agile projects and agile teams.

What is the definition of Exploratory Testing?

Exploratory Testing is a style (approach) of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution and test result interpretation as mutually supportive activities that run in parallel throughout the project.

Is Exploratory Testing a test design technique?

No. You can design tests in an exploratory or scripted way (to a degree each way). This is why it’s called an approach. But ET itself is not a technique (a way to group, design and interpret results of similar kinds of tests).

Does Exploratory Testing require charters?

No, but charters can certainly be helpful.

Does Exploratory Testing require a timebox?

No, but a timebox can help you create similarly sized sessions for Session Based Test Management.

Over time I hope to add to these.

Last Updated: 03/07/2022

Exploratory Testing Charters

An exploratory testing charter is a mission statement for your testing. It helps provide structure and guidance so that you can focus your work and record what you find in a constructive way.

How to Write an Exploratory Charter

My favorite way to structure exploratory testing charters is to base them on “A simple charter template” from Elisabeth Hendrickson’s awesome book Explore It!, Chapter 2, page 67 of 502 (ebook).

Explore (target) With (resources) To discover (information)

  • Target: What are you exploring? It could be a feature, a requirement, or a module.
  • Resources: What resources will you bring with you? Resources can be anything: a tool, a data set, a technique, a configuration, or perhaps an interdependent feature.
  • Information: What kind of information are you hoping to find? Are you characterizing the security, performance, reliability, capability, usability or some other aspect of the system? Are you looking for consistency of design or violations of a standard?

Examples of Charters

While this is my favorite way to structure exploratory testing charters (I think its a really straightforward template) it isn’t the only way. As a way to learn I’ve complied a list of example charters you can look at that can be found on my Guides Page.

A few examples include:

  • Explore input fields with JavaScript and SQL injection attacks to discover security vulnerabilities.
  • Check the UI against Apple interface guidelines.
  • Identify and test all claims in the marketing materials.

You can see some use the template and some don’t.

How do charters relate to Session Based Testing or Session Based Test Management?

Exploring can be an open-ended endeavor which is both good and bad. To help guide those explorations you can organize your effort with charters and then time-box those charters into “sessions”. You can use sessions to measure and report on the testing you do without interrupting your workflow; this is called Session Based Test Management.

You can use exploratory charters without using Session Based Test Management. I’ve seen many examples of people using Charters in JIRA stories as part of the testing criteria for sign off for testers, developers and product managers.

Review of The Selenium GuideBook: Ruby Edition

tl;dr If you’ve ever wanted to learn Selenium but didn’t know where to start, The Selenium GuideBook is the place (doesn’t matter which edition you use, it’ll be good).

 

Learning Selenium

The challenge of trying to learn Selenium WebDriver (simply referred to as Selenium) or any relatively new technology is that there is either too much disparate information (on the web), the material is out of date (many of the books I found) or of poor quality. With Selenium too many of the tutorials available to beginners had you using the Selenium IDE, which is a really poor option for writing maintainable and reusable test code. (See my previous post’s video for more on this.) I’ve walked out on conference workshops that sought to teach people to use the Selenium IDE to start their automation efforts. It wasn’t for me. I was going to do it right, I just had to figure out what that meant and where to start.

From the start I knew I wanted to learn about GUI test automation and more specifically Selenium WebDriver. I had tried WATIR (Web Application Testing in Ruby) and a few other tools but Selenium was open source and quickly becoming the go-to standard for automating human interaction with web browsers. It was and is the only choice.

Naturally I went searching the web for some tutorial or examples when I stumbled across several tutorials including Sauce Lab’s boot camp written by someone named Dave Haeffner. After struggling through the Bootcamp series (and finding some bugs in the process) I found Dave also produced a tip series called Elemental Selenium. I signed up for the Ruby weekly newsletter tips and went through many of the tips. Satisfied that Dave was worth learning from (good quality, relevant code examples) I decided it was time to try his book The Selenium GuideBook. I knew going into it, I was going to be the person maintaining the test suite and since I was more or less comfortable with Ruby I was happy The Selenium GuideBook came in that language!

Book Options

There are a few packages (book, code examples, videos, etc. ) for the language of your choice. As I said above I was more or less comfortable with Ruby so I ended up getting the “Ruby Edition, Just The Book” package. If I was doing this over today I probably would have done the “Cheat Sheets + Book” package and for JavaScript instead of Ruby.

The package itself contains a lot of great information and a number of materials:

  • The Selenium GuideBook; the Ruby edition is roughly 100 pages
  • Ruby code examples broken out by chapter
  • Elemental Selenium Tips eBook in Ruby
  • The Automated Visual Testing Guidebook

The first time I went through the book and code examples, it seemed redundant having different code for each chapter and example. It was only after I had gone through the chapters and examples for a second time, trying to apply and understand the differences that I began to understand the relative importance of seeing the code change chapter by chapter, example by example. The code examples all target an open source application Dave created called The Internet. It seems simple enough but many of the books and materials I went through either tried using Google or some badly written / hurried example.

The Book

Despite being less than 100 pages the Selenium GuideBook covers:

  • Defining a test strategy for your automation goals
  • Programming basics
    • Using page objects and base page objects to keep code clean
  • Locator strategies and tools
    • Relying on good locators seemed like a smart way to design tests. I wanted to avoid any record and playback tools and the poor locator strategies often employed.
  • How to write good acceptance tests
  • Writing Re-usable test code
  • Running on browsers locally and then in the cloud
  • Running tests in parallel
  • Taking your newly built tests, tagging them and getting them loaded into a CI server.

The whole package literally. In the preface Dave says the book is not full and comprehensive, it’s more of a distilled and actionable guide. The book really shows you how to start thinking about and building a test suite. It’s up to the user to take what they learned here and apply it to their application. That’s the fun part of the Elemental Selenium tips as well.

Applying the Book and Examples

After I had gone through all of the chapters and examples once, I went back through the relevant to me chapters and examples doing the following:

  1. Start with some very simple login tests.
    1. The book starts out this way as well. Writing tests that are not DRY or re-usable. but eventually get that way.
  2. Continuing through the code examples, getting a little more complicated and applying it to my own application.
    1. As I built out tests and start to see commonly repeated patterns, abstract out repeated code into relevant page objects. Eventually getting to a base page object.

In hindsight the hardest part of applying the book was trying to understand and apply a locator strategy within our application. While The Internet target application is great, it’s also a bit simplistic. Good for teaching, hard for bridging the sample application to the target application. Our single page .NET application was far more complicated and it took several attempts before I understood how my own strategy would work.

The transfer problem is always difficult. I mean how do you take what you learned and apply it to a slightly more sophisticated problem? It’s a difficult problem, not really a criticism of this book. It’s worth noting that whenever I had questions about what was written in the book, found a bug or two, or got stuck I could email Dave and within a week or so get a helpful response back.

Mission Accomplished!

With the lessons in this book and the Elemental Selenium Tips I was, through some focused time and lots of iterations, able to get a fairly good set of automated acceptance tests running against our application.

In other words, I highly recommend you buy the book. It’s slightly more expensive than similar books about Selenium but it’s far more effective. You are also directly supporting the author and with free updates and email tech support I think its well worth the cost.

Don’t believe me? Watch this video of Dave giving an overview of the GuideBook:

Additional References:

  1. http://www.tjmaher.com/2015/06/spotlight-on-dave-haeffner.html
  2. http://knowledge-anxiety.blogspot.com/2015/06/the-selenium-guidebook-and-thoughts-on.html
  3. http://davehaeffner.com/works/
  4. https://seleniumguidebook.com/

Exploratory Charters in GitHub

GitHub Issues
GitHub Issues (for TestingConferences.org)

Since CAST 2015 I’ve wanted to implement an interesting idea that could potentially give my testing greater visibility and greater scrutiny: Putting exploratory testing charters into our project tracking tool.

At work we use GitHub to host our code which means we use GitHub Issues as our bug tracker. Then we use ZenHub on top of GitHub Issues as project management / tracking tool. As the sole (omega) tester on the development I use GitHub issues for a number of activities:

  • Filing and reviewing issues (bugs and enhancements)
  • Occasionally reviewing pushed code changes referenced in the issues I’m tagged on
  • Occasionally reviewing pull requests
  • Committing the rare change to our production code (I’ve done this once so far)
  • Committing code changes to our test automation repo

Do charters belong?

Putting my exploratory charters into GitHub Issues (so they’d be visible in both GitHub and ZenHub) seemed like a worthy proposition. On the one hand it would allow my charters and subsequent notes to be seen and commented on by the whole team. On the other hand, would they really belong? GitHub Issues can be used to track tasks, bugs and enhancements, it’s a pretty simple and straightforward difference. My teammates have been creating tasks for a while and in the last few months I’ve been adding charters.

So far, no one but me seems to care. Great! First step complete. I already break the charters up into test sessions with notes and debrief information. After each session that information usually stays in my session notes. Instead it should flow into it’s corresponding charter’s task so I have a stream of updates as to what this charter has uncovered and what there’s still left to do.

That’s my next step, use the GitHub task to more actively record the details of the session and report back how the charter is going. From there I can keep track of the amount of effort I’ve spent (ZenHub allows us to track hours assigned to a task) per charter. Once this becomes more comfortable, the final step is to get our project management team to help influence the prioritization and maybe creation of those charters. Small steps! Shout out to Jesse Alford for the suggestion!

And nothing else funny happened

I was recently talking with someone about their testing strategy and process when I noticed they were trying to build overly-detailed test scripts (procedures). It didn’t take them long to realize specifying to such detail often left them bored (writing became redundant) and so each test became less and less detailed. I offered to take a look at their tests to see if I could help improve things and as I saw, what I consider to be, their “typical” scripted tests with each line having a step and expected result I started thinking of something Pete Walen once said:

…and nothing else funny happened.

Peter Walen, STPCon

Pete Walen dropped this gem at STPCon in San Diego a few years ago during Matt Heusser’s talk Where do Bugs Come From? Matt had just shown the Moonwalking Bear video:

I’d guesstimate at least half of the packed room hadn’t seen this. The discussion turned to scripted testing and the inherent problem of in-attentional blindness. Pete shared a past experience where he told some testers he was working with to include the phrase in the expected results of their scripted tests. It’s a kind-of-heuristic to help someone remember, even though they are focusing on one thing at depth, they need to be aware of the potentially many other things going on so that “nothing else funny happens”.

It was such a simple, powerful idea it continues to stick with me, especially when I see someone trying to specify “most everything” in their scripted tests.

As it happens, this post coincides with STPCon 2015 back in San Diego right now. There looks like a lot of awesome speakers (Matt Heusser, Michael Larsen, and Smita Mishra) but I’m particularly intrigued by Andy Tinkham’s High Volume Automation in Practice and Dave Haeffner’s Using Selenium Successfully. I wonder, is anyone live blogging?

Review: Exploratory Software Testing – Tips, tricks, tours and techniques to guide test design

Some of the first testing books I read were from James Whittaker’s How to Break Software series. Those books, like this one, are laid out in a practical manner with each chapter focused on a specific attack or approach making them easy to read, reference and apply. Perfect for learning. I picked up this book a few years ago when I started questioning the way I was testing. The material was new to me and made me ask what is exploratory testing and what does touring have to do with it?

According to Whittaker (pg. 16) exploratory testing (E.T.) is testing where scripts or rigidity have been removed (paraphrasing). Whittaker explains his terms “E.T. in the small”, decisions made where the scope of the testing is small and “E.T. in the large”, decisions made when the scope of testing is large (small might be a screen in an application while large is the whole application). At the end of chapter 3 he mentions E.T. can be done in a way that allows test planning and execution to be completed simultaneously which is one of E.T.s most important aspects and simplest definitions. Touring (as in a tour guide or sight-seeing) becomes a metaphor for and a way to structure E.T.

There are eight chapters in the book plus a number of appendices. In the first few chapters Whittaker discusses what he sees as the case for software quality (the context of the book), introduces E.T. and explains how he uses it, in the small and the large. The next four chapters cover tours he and others have come up with. The last chapter is about how Whittaker sees the future of testing or at least how he did at the time of publishing.

The first appendix, A, is one of the most important parts of the book: building a successful career in software testing. Whittaker talks about how he got into testing and gives some advice on “getting over the hump” to be a better tester. Appendix A is short but worth reading. The rest of the appendices are old blog posts from his Microsoft days.

As a beginner I found this book much more valuable than I do now several years later. I understand E.T. is an approach to testing that can but doesn’t necessarily include tours or scripts. It isn’t just manual testing either. For reference Michael Bolton (the testing expert) has some good posts in what E.T. is not: (notice how the first post is about touring?)

As you might not guess from the title of this book it does not do a proper job explaining E.T. in a way that one can use it, aside from following the tour metaphor. In fact after reading it again this book seems to say to the reader: these tours are the best, don’t you agree? It’s important to understand exploratory testing is about the way you work, and the extent to which test design, test execution, and learning support and reinforce each other.

According to James Bach the term “exploratory testing” was coined and first published by Cem Kaner and has been worked on by Bach, Whittaker and Kaner (among others) over the last decade. It seems a bit odd that in a book about E.T. Whittaker never mentions their work and provides no references for the reader to follow up. Apparently Whittaker thinks the easiest way to explain E.T. is through testing tours (hence the book) while Bach has a more direct explanation of what constitutes exploratory testing. I found Bach’s post more informative, applicable and frankly cheaper than Whittaker’s Exploratory Software Testing book.

Exploratory Software Testing (the book) offers a limited metaphor for understanding exploratory testing. It isn’t as practical as Whittaker’s previous books because you can’t apply the teachings very well without fully understanding what E.T. is and how tours fit in. If you only want ideas on how Microsoft’s testers used the touring metaphor to “perform” exploratory testing then you’ll get four chapters of information otherwise Exploratory Software Testing is worth skipping.

I wrote the same review on Amazon under the heading “Limited metaphor for exploratory testing”.