uTest’s Business Model

Note: I am an active uTester.

The last few months I’ve completed a number of rounds of testing for uTest’s clients, mostly in dealing with web applications for my iPhone. In fact a majority of work I’ve done since joining has been for functional testing of mobile applications. It’s been fun because mobile testing isn’t in my area of expertise but is a nice break from my normal routine and I like learning new things.

uTest’s Business Model:

A few months ago I was talking with my boss about new options for helping me test our software. I work for a small company where I’m the only tester. Often the backlog for getting our releases out is me. My boss was talking about adding an offshore resource and I brought up the idea of uTest and their crowdsource model. He thought it was an interesting idea and so he contacted uTest to get more information.

A few weeks later we had a quote from uTest and a chat with one of the sales reps which gave me a interesting perspective into their business model. uTest prefers to sell their services in packages which generally include several rounds of testing (the time between rounds is up to you). The sales engineer’s try to get an understanding of your testing needs and then give you a flat price per round with a minimum number of rounds plus a monthly Software as a Service (SaaS) charge for access to uTest’s application – a must have for the tester’s to submit bugs, test cases, etc. I think our application was considered pretty big / complex so for 3 rounds it was just north of $7,000.

That means we’d pay uTest $7k upfront plus the SaaS access charge each month. From there we’d work with a project manager and tell them how many tester’s we need, what type of backgrounds tools they need, etc. Then the project manager builds the test process and plan with you. Essentially you are hoping you get a good project manager otherwise the money you drop and the test outcomes may not be worth it. The actual payments the tester’s receive (for test plans and bugs) comes from uTest out of that flat fee.

Makes me wonder what the average payout to tester’s are per round of testing? Probably less than $1k depending on the size. That means a majority of the money is going to pay for your project manager and to uTest’s wallet.

5 Ways to Revolutionize your QA

I can’t remember where I originally found this post and the corresponding eBook but the eBook is definitely worth taking a look at. Here is the former uTest blog post, now Applause blog post.

The 5 ways or insights are:

  1. There are two types of code and they require different types of testing
  2. Take your testing down a level from features to capabilities
  3. Take your testing up a level from test cases to techniques
  4. Improving development is your top priority
  5. Testing without innovation is a great way to lose talent

In point 2, James Whittaker also talks about a planning and analysis tool he used at Microsoft called a CFC or Component – Feature – Capability analysis. This allowed them to take testing down from features to capabilities.

The purpose is to understand the testable capabilities of a feature and to identify important interfaces where features interact with each other and external components. Once these are understood, then testing becomes the task of selecting environment and input variations that cover the primary cases.

While this tool was designed for testing desktop software I’m inclined to think it would work well for testing web applications. Essentially with the CFC you are mapping out the individual components / features in the web application in a branching form that closely resembles a mind map. Matter of fact a mind map might be better! =)

Secrets of a Buccaneer-Scholar by James Bach

I recently finished reading James Bach’s book Secrets of a Buccaneer-Scholar. I purchased the book mistakenly thinking it was a book on software testing (I didn’t really read the synopsis before buying it) but was pleasantly surprised after having read it.

I’d heard Bach was an expert in software testing, checked out his blog and then found this book online:

At first I wasn’t sure what to make of this book but it gets better as it goes on. Just as the title says it’s about how the pursuit of Self-Education can lead to success based on the author’s (James Bach) experience doing just that in the field of Software Testing.

I considered posting a book review here but I found this review on Amazon with which I agree. Now all I need to do is read his book Lessons Learned in Software Testing.

Designing a Web App

This is a really cool video from Ryan Singer who demonstrates how he and his company (37 signals) design web applications.

Ryan Singer at Future of Web Apps, London 2010 from Ryan Singer on Vimeo.

They follow a few simple steps to make it all happen: model, screens, designs, html/css and live code. In his demonstration, Ryan makes it look incredibly easy to get something up and running. It’s worth a watch if you want to learn to how rapidly design and prototype a web application.

STAR West 2011

It’s official I’ve registered for STAR West 2011 (also know as Software Testing Analysis and Review for the west coast) in Anaheim, CA. I’m only going for Monday and Tuesday, the tutorial days, but I’m excited for the ones I’ve chosen:

A Rapid Introduction to Rapid Software Testing with Michael Bolton. It’s a full day course. Hopefully it’s interesting so I can stay awake the entire time! http://www.sqe.com/StarWest/Tutorials/Default.aspx?Date=10/3/2011

The quality of the courses on available on Tuesday is far below Monday’s so I went with two half day classes. In the morning I’m taking Using Visual Models for Test Case Design with Rob Sabourin. In the afternoon I’m taking Testing Web-based Applications: An Open Source Solution with Mukesh Mulchandani. I’m hoping it will broaden my understanding of automation since the full day automation tutorial from Monday isn’t available. http://www.sqe.com/StarWest/Tutorials/Default.aspx?Date=10/4/2011

James Whittaker from Google will be there Monday morning as he mentions on Google’s Testing Blog here. Google has two people presenting on Monday: James Whittaker in the morning talking about How Google Tests Software and Ankit Mehta on Testing Rich Internet AJAX-Based Applications.

If I had more time I’d check those two tutorials out but I don’t. Bummer. Hopefully Google’s Testing Blog will recap some of the things they covered.


GTAC 2011 and STARWEST 2011

The two big Software Testing events of the year, GTAC or the Google Test Automation Conference, and STARWEST are both being held in October of this year. Big month for software testers!

According to the Google Testing Blog GTAC 2011 will be held in Mountain View, CA during the week of October 25th. STARWEST 2011 will be held in Anaheim, CA during the week of October 2nd.

The real question is how do I get my company to pay for both? Hopefully GTAC is reasonable inexpensive.

Summary: How Google Tests Software

As a software tester I try to learn as much as I can about how other companies test software. It just so happens that through Google’s testing blog James Whittaker has taken steps to outline just how Google does it.

If you’re interested in learning more I’d recommend reading through the five part series by going to the Google Testing Blog directly but feel free to check out my summary and the things I found interesting:

Google’s organization structure is such that they don’t have a dedicated testing group. Instead the company has more of a project-matrix organizational structure where testers are located in a group called Engineering Productivity where they report directly to a manager but are then shared to individual product groups like Android, GMail, etc. Through this they are able to move around to different groups inside the company based on a particular project and stand to gain a better experience. Engineering Productivity also develops in-house tools, maintains knowledge disciplines in addition to loaning out engineers.

Google has a saying: “you build it, you break it”. They have essentially 3 engineering roles Software Engineers (SWEs), Software Engineers in Test (SETs), and Test Engineers (TEs). SWEs write code, design documentation and are responsible for the quality of anything they touch in isolation. SETs focus on testability, they write code that allows SWEs to test their features, refactor code, and write unit testing frameworks and automation. SETs are responsible for quality of the features. TEs are the opposite of SETs and are focused on user testing. They write some code in the form of automation scripts and usage scenarios and coordinate and test with other TEs. These descriptions are a bit over generalized but you get the idea.

It’s interesting to note that in all of the companies I’ve worked for the SWEs and SETs are the same people and TEs are usually focused on the low hanging fruit. Instead Google blends development and testing to prevent bugs / lapses in quality instead of trying to catch them later when it is more expensive and harder to fix.

As a rule Google tries to ship products as soon as they provide some benefit to the user. Instead of releasing new updates / features in large releases Google tries to release, get feedback and reiterate as fast as possible. This means less time in-house and more time getting their customers responses. Yet in order to get out to production Google has 5 Channels to get through: Canary, Dev, Test, Beta and Production. The Canary channel holds experiments and code that isn’t ready to be released. The Dev channels is where the day to day work gets done, the Test channel is used for internal dog fooding and potential beta items. The Beta and Production channels hold builds that will get external exposure assuming they have passed applicable testing / real world exposure.

Finally Google breaks down their types of testing into three broad categories that include both manual and automated testing: Small Tests, Medium Tests and Large Tests. Small tests are written by SWEs and SETs and are usually focused on single functions or modules. Medium tests involve two or more features and cover the interactions between those features. SETs are mostly responsible for Medium tests. Large tests are three or more features and represent real user scenarios as best as they can be represented. The mix of manual and automated testing depends on what is being tested. James reiterates it’s not how you label the tests just as long as everyone in the company is on the same page.

And there you have, roughly, how Google Tests Software. You can see they spend a great deal of time working on preventing bugs from ever coming up so they can focus their Test Engineers on bigger potential problems and less on the low hanging fruit – which completely makes sense. Now how you and I apply these things to our own testing framework is the real challenge!

Windows Time Sync error: 0x800705B4

When trying to display the time difference between a local computer and another time source using the Windows Time Sync command: w32tm /stripchart /computer:targetcomputer /samples:number /dataonly you may see the response as “timestamp, error: 0x800705B4”. This just means the local machine’s time source isn’t available.

To fix this error you need to set the client machine to use an external time source like another server. In order to do that the other server must be setup as a Authoritative Time Server. Then configure a manual time source using w32tm /config /manualpeerlist:targetcomputer /syncfromflags:manual /update.

When running the manual time source command on a Windows 2008 R2 SP1 machine I got a different error “The service has not been started. 0x80070426”. To fix this problem go to services.msc, find Windows Time and start it. When you rerun the command all will be well.

I’m not sure why Windows Time Syncing has to be such an issue without a domain controller setup but it is!

Associate SQL Server user with login after db restore for SQL Server

This post was borrowed from Computer Cabal and expanded upon:

After you’ve restored a backed up SQL Server database instance you may find the user logins are no longer associated with the users. You can’t make this fix via SQL Server Management Studio but you can run the commands below to fix it. Note: It only works for SQL Server 2005 SP2 and later, so the first thing to do is check what version of SQL Server you have.

    1. Log into the Management Studio for SQL Server 2005 run:

SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

  • Or if you are using SQL Server 2008 run:

SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition')

  • In the second column you should see a table with a row name you should see “SP2” or “SP3”, if you see something like “SP1” or “RTM” or don’t see a second column then you need to upgrade. Here’s the link to download SQL Server 2005 SP3.
  • Once you’ve got at least SQL Server 2005 SP2 run:

alter user [user_name] with login=[login_name]  

  • Now you should be all set.

For more information on How to Identify SQL Server versions and editions go here.