My Testing Philosophy

As I study software development, especially testing, I’ve begun to develop certain views and a general philosophy on testing. In this article, when I say philosophy I mean:

[T]he critical study of the basic principles and concepts of a particular branch of knowledge, especially with a view to improving or reconstituting them.1

Or simply:

[A] foundation for studying and improving my ideas on software testing.

I’ve generally been aware of this development (others have highlighted it during discussions) but when I saw Andy Tinkham go through the exercise of writing out elements of his philosophy, I wondered if I could do something similar?

Then I created my list.

I borrowed heavily from Andy’s and I certainly haven’t covered everything but it’s a good start. Some elements are bound to change with time (as both I and the industry learn better) but here’s what I believe about software testing (and to a greater extent quality):

  • Software is eating more of the world.
    • Thus, quality is becoming more complex and important.
  • Quality is subjective. What one person values another might not.
  • Quality is everyone’s responsibility and can’t be added after the fact.
  • Software needs to solve the customer’s problems or it is useless.
  • Testing is a challenging intellectual process
    • Thus, any testing activity that discourages thinking or questioning while performing it is potentially harmful. Test execution off of pre-written scripts is often an example of this.
  • There are no best practices, just good practices for a particular context. Blindly applying a practice to a situation because it worked somewhere else can cause more harm than help. Understanding the context of any practice will be used in is crucial.
  • Complete testing is impossible and therefore involves tradeoffs.
  • Testing is all about providing information to the team and stakeholders. If testing is failing to provide the information that is needed, it is wasting time and resources.
  • To provide that information, test reporting needs to clearly tell 3 stories: the story of the product quality, the story of the testing done and not done, and the story of the testing quality (why what was done was or wasn’t sufficient).
  • A test is an experiment designed to reveal information (or answer a specific question) about a product or service.
  • To be successful, testers need to approach a SUT trying to dis-prove the existing understanding (for example, that it has bugs) rather than proving some functionality is correct.
    • Otherwise, we run the risk of falling victim to confirmation bias.
    • It’s all about empirical evidence.
  • Tests are only ever as good as their oracles.
  • Testers are not the gatekeepers of the quality. They should have a voice, but not the sole voice, in release decisions.
  • The value of any action has 3 components: The benefit gained by completing the action, the costs incurred by performing the action, and the hidden costs of not getting the benefit from performing other actions which can no longer be performed. Choosing the right action at any point in time means striking a balance among these 3 elements.
  • Writing out detailed, explicitly defined tests is rarely the best use of time and in many cases it’s improbable (to fully detail them).
    • Letting the tester executing tests make decisions that don’t impact the test’s goal leads to better testing.
    • Writing down every expected result can be a huge amount of work on one testing task and often doesn’t consider what other testing tasks remain.
  • Writing tests at the start of a cycle of testing effort means we’re creating those tests at the point where we have the least knowledge about what we’re testing that we’ll have during the cycle.
  • Communication is essential.
  • Variety in tests, techniques and approaches is essential.
  • Conventional software testing metrics are often misleading and can lead to undesired behavior. However, they’re what we have to work with at the moment, so until something better comes along, we need to use them to approximate the information we need. We just need to ensure we use them wisely and be conscious of the impacts and risks in doing so.
  • Bug reports (the tester’s primary work product) should be written in a clear, concise and persuasive way that convinces the reader to take some action.

Thanks to Andy for the inspiration and I highly recommend testers try this exercise. Think this was cool and looking for another exercise? Try thinking about your commitments to your programmers or your team and write them out like I have.

The challenge for me will be keeping this list up to date and constantly evolving. Think you can help? Feel free to challenge one or more of these elements, you’ll only be helping!

(Last updated on 12/21/2015)

  1. Dictionary on