I recorded and submitted my 7 minute talk on “Using Test Idea Catalogs for Better Testing”. The premise is:
Testers can develop a set of tests or test concepts for a specific object or risk and re-use them in similar projects or products. Catalogs come in many shapes and forms, they can be lists or more detailed. They can be public or private. They can be developed by individual testers or as teams within companies. But they all help you test better!
Inspired by Justin Rorhman’s post of a similar name with a slight twist focusing on tools that generally help accelerate my testing.
As a test and quality specialist embedded in an engineering team I have a lot of work to do on any given day. Our engineering team’s goal is to ship quality software on a regular basis to deliver value to our customers. Naturally I rely heavily on a number of tools to help me better understand what is going on with the software or code under test and/or to accelerate what I’m able to do in a given amount of time.
Like Justin said in his article “Testing tools don’t make me a good tester, but they do make me a better tester. These tools help amplify things I need to do repeatedly or give me access to the product I wouldn’t have otherwise. They also shape the way I think about testing problems, for better and worse.”
In no particular order these tools are:
GitHub. We use the GitHub workflow to deliver all of our code including application, library, and test automation. With each push to a particular branch our CI system builds the application and runs our unit and integration tests. With each Pull Request I can see all or just specific changes, code reviews and test run summaries. All this helps keep our build pipeline clean, makes it easier to identify potential issues quickly and helps me quickly build my test strategy.
Docker. Each separate application of our system has been dockerized (aka runs in a container). This makes it incredibly easy to set new people up (a few hours of setup instead of a day) and gives us some powerful logging for each application. It also reduces conflicts when switching between branches where libraries or decencies have been changed.
Heroku review apps. As I mentioned above with each push our application is compiled through our CI system. Once finished our CI system pushes that complied build to Heroku which automatically provisions a staging like environment accessible through its own url. (This feature is called a Heroku Review application). In about 5 minutes after a set of changes have been built our team now has a brand-new test environment ready for those hard to test local problems like testing on real mobile devices or a few special integrated services. Being able to build these new automatic quickly allows our developers to show off / get quicker input to the work they are doing.
Browser DeveloperTools. I primarily use Chrome and Safari but the developer tools in each browser are priceless. From Inspecting requests and looking at their data, to setting or removing experiment variables, debugging problems, looking for problems, profiling, responsive design, layout, checking DOM elements, etc browser DeveloperTools really are the first place to start. It would be so much harder to learn what is going on without them.
CircleCI. Great for keeping your deployment pipeline clean, running your build tests and ensuring those changes get where they are going each time. It doesn’t much matter what CI system you use it’s helping to accelerate something that would normally take away from developer or tester time.
EverNote. Surprising or not, I’m always taking notes about any number of things like quick terminal commands, operations for creating data, migrating databases in heroku, setting up configurations, test ideas, questions for follow up, etc. Anything I might forget goes here and I can get to it anywhere, any place on any device.
iTerm2. A command line replacement utility with Oh My Zsh installed that I’ve customized and continue to customize with shortcuts and aliases to make my repetitive tasks quicker. Also customized with visuals showing which repo I’m in, what branch I’m on and if there are uncommitted files, because I do enough on the command line.
Automated UI acceptance tests. Running our acceptance tests for each staging deploy (or set of deploys) allows us time to do more deep exploration on the application as a whole with a reduced focus on those areas covered by the existing tests.
Part of the desire for writing this was to record what my work looks like now and then be able to compare it later-on in the future. However, in the time since I started writing this article I changed jobs and now the future is here. Well, kind of. My new team’s development stack is a lot more varied, which means so are the tools I’m using. I think it’s still best to wait for half a year or so before I write a follow up post on the tools accelerate that work. In the meantime, what tools do you use that accelerate your testing?
On April 11, 2018 I gave a workshop called Contributing to GitHub is For You is For You (abbreviated as C2GI4U) in Des Moines, IA to the Des Monies Area Quality Assurance Association (abbrevitated as DAQAA). Roughly 36 people signed up for the DAQAA event meetup and of those about 22 showed up. Scheduled for 3 hours (including a break in the middle for food) I set out to cover a number of topics around using Git + GitHub including:
Submitting changes in Pull Requests
Merging, reverting, cloning, forking, and other common commands, etc.
Using git locally from the command line and GitHub remotely through the web
In a little less than 3 hours we were able to run through three quarters of the exercises, covered all of the basic git commands and many of the intricacies of using the Git + GitHub workflow. Everyone walked away knowing how to grab and contribute to a project in Github using Git in the command line. A few people were overwhelmed having to pick up terminal commands in addition to Git commands, but most were able to get through the exercises and commit jointly to the C2GI4U repository. By the end a few felt so empowered they were going to apply what they learned the next day at work. (This point made me feel quite well.)
Workshops (generally) are fun because we get to do hands-on technical things. They also bring up interesting challenges, some of which are expected and others not so much. For example I expected some people might have to learn to navigate the command line (or re-familiarize themselves). Although we split time between GitHub’s nice UI you can’t get away from the command line but with practice during the workshop many people became more comfortable. However there were a few things that I didn’t expect and hope to improve upon the next time:
Provide a list of git commands we will use alongside the exercises. I had a list of all the git commands we were using as well as a list of exercises BUT they were in separate places.
I included a few cheat sheets with a bunch of different git commands but these were too broad to teach from, especially for people brand new to git.
Include information about storing GitHub up credentials. Although this wasn’t a big issue (you get a prompt to login) I always forget this step since my credentials are saved in my terminal. Everyone ran into this problem.
Make sure people are aware of the location of their code. I was good about this up front, but I need to make it a part of the pre-requisities and slides. This is as simple as making sure you are in your Documents folder before cloning a repo.
Branching can get quite complex. I’m also not sure how much of a challenge this was for people to understand and perhaps because of this how exactly to address this complexity except by going through exercises (I even drew a few diagrams on a whiteboard). Ideally I’d have more information / follow up references and maybe another diagram or two. I don’t think I explained Remote vs local branches as well as I could have.
Not really an improvement but a general note of thanks: A few of the organizers from DAQAA had experience with git + GitHub and so I was happy to have had a few helpers roaming around. An unexpected but happy side effect was participants helping one another out which made my job that much easier.
The goal here was getting closer to the development workflow (like git + GitHub does) to help reduce the knowledge gap for testers, business analyst, other developers, etc. and help make them more comfortable with the technical aspects of software development. (There were a couple of attendees who were learning to program in Java and the course they were taking assumed they knew how to use Git + Github which they did not. This class helped them become more proficient.) Overall I think we achieved that goal.
On June 13th Matt Heusser and I will be giving a talk at the Online Testing Conference on the value for testers in contributing and using GitHub. Actually it will be more than just a talk, we’ll do a full demo on how to get started on GitHub by creating and contributing to a repository of your own. Then we’ll give you some ideas on how to use it going forward. (Hint: there are lots of uses besides code.)
Aside from it’s many awesome lists GitHub is a really good place for open source testing tools, libraries and frameworks (and their corresponding code). I’m pleasantly surprised by these new (and sometimes old) testing resources, so I’d like to highlight many of them in the hopes others might also find them useful.
The challenge with listing any GitHub projects is that there are really too many projects to list. I’ll push this problem and mention a few here anyways. Many of these projects are Ruby based because that’s my language of choice. I’m sure there are similar projects for your favorite languages as well.
Many of these tools are easy to download or install without any knowledge of their underlying code. A few like The Internet or Ally will require a little more work.
BugMagnet. Both a Chrome and Firefox extension that contains lists of values you can use for boundary testing. Great exploratory testing companion.
Form Filler. Other Chrome extension is great for filling in standard forms (login pages, etc).
The Internet. A test project for playing with selenium automation but could probably be cloned and used for other testing / practice purposes.
AutoHotkey. A windows desktop automation project for helping you speed up your workflow.
These offer extended functionality, generate data or generally make testing easier.
Faker. This great gem will help you generate fake data for testing. Need passwords, usernames, email address, city information? Faker has it.
Parallel Tests. This gem is for running multiple tests at one time. Personally I use this for running concurrent automated tests locally and at Sauce Labs.
Mailosaur Ruby Client. This gem is part of a mailosaur service but in theory will help with end to end automated testing around mocking email services.
BrowserMob Proxy. A proxy for monitoring and manipulating traffic. ElementalSelenium has a few tips showing how to use BrowserMob in your test automation.
These are pretty well known across the testing landscape and for that reason are worth mentioning. Many of them have good wikis, documentation or other related resources you can learn from in addition to the underlying code and bug trackers. YMMV:
My goal with sharing these is to point on the many very good tools available for testing and I hope I’ve accomplished this. Did I miss any projects that you find useful for testing? If so leave a comment and tell me what you like about the particular projects!
I spend a lot of time on GitHub and it can be a great place for finding open source libraries, tools, frameworks and pretty much anything else you might want to version control. This includes lists (and more often than not, lists of lists). The challenge is finding just those lists that contain value and not chasing around each individual list of list in a recursive never ending search.
Over time and when looking for certain types of failures (or bugs), patterns emerge. Some of these patterns can be captured as data, like password dictionaries or image catalogs, or as a collection of test ideas. Some authors have made lists this with heuristics (Bach, Bolton & Hendrickson) while others have published lists of failures common to certain applications or languages (Kaner).
Here are 9 lists I’ve found to be good references when testing.
Useful for boundary analysis and equivalence class partitioning, input field catalogs are basically collections of values you can use to try to trigger failures based on the data-type of the input field. For this reason they are often broken into specific data types like Strings, Integers, Floating Points, etc.
Other valuable lists that don’t easily fall into a single category:
My own catalog of images. Based on size and format you can use this catalog for testing image uploads. Searching Google I wasn’t able to find any specific collection for testing, so I made my own.
Awesome Test Automation. A curated list of test automation frameworks, tools, libraries, etc. The list is pretty good. I use Ruby and they had a good list of Ruby gems for generating test data.
TestingConferences.org. A simple list of software testing conferences and workshops published collaboratively with the testing community.
Free Software Testing Books. That’s right, a collection of free software testing books. Although some of these appear to be papers, guides and “demo” chapters, it’s still a good (cough, free) reference!
Did I miss any lists that you find useful for testing? If so leave a comment and tell me what you like about the particular list!
Since CAST 2015 I’ve wanted to implement an interesting idea that could potentially give my testing greater visibility and greater scrutiny: [highlight]putting exploratory testing charters into our project tracking tool.[/highlight]
At work we use GitHub to host our code which means we use GitHub Issues as our bug tracker. Then we use ZenHub on top of GitHub Issues as project management / tracking tool. As the sole (omega) tester on the development I use GitHub issues for a number of activities:
Filing and reviewing issues (bugs and enhancements)
Occasionally reviewing pushed code changes referenced in the issues I’m tagged on
Occasionally reviewing pull requests
Committing the rare change to our production code (I’ve done this once so far)
Committing code changes to our test automation repo
For the past few years one of my professional goals has been to attend (at least) one testing conference or workshop per year, mostly because it’s such a great way to recharge and learn what other practitioners are doing. Unfortunately, I couldn’t find a good source of events aside from talking with others, so I pulled together a list on my own. The value of any events list is only as great as the number and quality of events listed so I’ve open sourced the list and posted it online with the hope that it becomes driven-by and representative of the community as a whole (as opposed to what I might like or prefer).
Introducing TestingConferences.org – a simple list of software testing conferences and workshops published collaboratively with the testing community.
I hope this site is or becomes a useful resource for the software community as a whole. I also hope others will help by contributing because that’s the only way it will become better or at least maintain its usefulness. It doesn’t matter if you are a vendor or just a fan of workshops and conferences, please add to it!
A little more detail
I’m not sure how I found my first conference, StarWest, back in 2011 but I did. Every other conference or workshop I’ve been to since has been a referral or recommendation by others, including:
The Rapid Testing Intensive workshop (RTI)
The Workshop on Teaching Software Testing (WTST 13)
STARWest (I’ve been to this twice I think)
I say referral / recommendation because the primary way this stuff was and is communicated are through the people who’ve attended. Sure you can Google “software testing conferences” or come across an advertisement in a testing publication but those are only useful if you have an idea of what you are looking for. Even if you do those things you might come across the most advertised, most popular but not necessarily anything near you (especially if you are outside the US) or relevant to your particular tastes. To me, that’s sad.
Conferences and workshops are great tools for conferring, collaborating and learning. At least part of your decision to attend a conference or workshop is determined by knowing what conferences are available and where they are located. That’s where this list comes in. There was never any single source of active conferences and workshops; especially with workshops you had to be in-the-know, else just stumble across one that was occurring.
Now I’m asking for your help so we can publish this list collaboratively within the testing community.
Interested in what conferences and workshops are considered eligible?