Outsourcing Web Site Testing

A company might consider outsourcing a web site's testing for several reasons.

  • The company may be launching a new site, or a new version of a site, and may feel that most testing tasks can be relegated to a contractor.
  • The company may not have the resources -- people, skills, software, hardware or time -- to perform the testing themselves.
  • The project to be tested may be of such as short life span that the company doesn't need any long-term investment in testing processes for the project.
  • The company might want an independent third party to perform the testing, in order to get a more objective view of site quality.
  • The company may even be outsourcing the development and coding for the site, making the outsourcing of the testing a reasonable decision. (Even more reasonable would be a firm that provides the coding and the quality control for its own code.)

The decision to outsource testing needs to be a well-considered decision, because most people who aren't responsible for the testing misunderstand the meanings and scopes of the concepts involved in testing. Two very important issues must be resolved before taking the outsource step: first, the company that owns the web site must be absolutely clear about the scope of the job -- including the tasks, processes, and responsibilities -- that they want to hire out; and second, the company must be sure that they are speaking the same "language" as the test firm they will hire.

If I want to outsource any testing tasks for my site, I need to understand clearly the nature of the tasks, and how the tasks fit into the overall quality plan for the site. Many testing tasks can be parsed out to contractors; for example, hiring another firm to perform usability reviews makes excellent sense because many firms specialize in usability testing. Usability testing, however, is not the same as quality assurance, and if I hire a usability firm under the assumption that my quality assurance needs are thereby being handled I will be making a very big mistake.

If I want to outsource quality control for my site, I'm making certain assumptions about how I will do business. Quality control is a system for testing all output against a standard of quality; this is an ongoing process, because in a proper quality control environment all output has to be tested. If I hire a firm to help me test prior to launching a site, I need to either form a partnership with the test firm to help with future production or build my own team of quality control testers. Neither option is necessarily difficult, but I have to consider this before making the decision to outsource because the choice will affect my future workflow.

If I want to outsource quality assurance, I'm making a decision to introduce the test firm deeply into my company's decision process. Quality assurance is a proactive process that requires close proximity to all phases and aspects of the creation and maintenance of a web site. If I exclude the role of quality assurance from important meetings and discussions -- for example, design meetings -- I'm crippling the process. Quality assurance is an order of magnitude more complicated, more involved, and more important than testing of quality control alone, and assurance requires constant attention and a penetrating interest in improving everything. This understanding must play a role in any discussion of outsourcing quality assurance.

I do think that a skilled and dedicated testing firm can be a vital part of a quality assurance effort, as long as the process, and the relationship, are managed by somebody on the web site's team who is ultimately responsible for the process. Outsourcing testing tasks is a great idea, especially if it frees up resources -- time and attention -- for the web team to focus on higher-level issues. I'd gladly pass off browser compatibility testing to have more time to work out architectural issues, for example.

Issues to Consider When Outsourcing Testing

I spell out some points below that I think are important to address when making the decision to outsource testing for a commerce site. This is a big decision, and approaching the decision itself from a quality assurance point-of-view shows that outsourcing testing is not simply a matter of jobbing out a set of simple tasks.

Understand the scope of what you will outsource.
Look past the buzzwords and learn what testing, quality control and quality assurance really mean and involve. Learn about the importance of these concepts to your site's and your business plan's success. When you understand the scope of the task, you can better manage your expectations -- if you pay for browser compatibility testing but consider it to be a quality assurance program, you will be disappointed on several levels.
Who is writing your code?
The source of your programming code is something to consider, because fitting a contractor into your code development processes may be awkward. If you contract out your coding -- for example, say you have a "technology partner" from whom you buy your site code -- you need to get the code from the developers to the test firm, and possibly through several iterations of this cycle, before the code is released. If the testing is contracted out, you distance your company's decision makers from close-up experience with the code, which will be a problem if the quality of the code is such that you must modify your project plans.
Testing is not quality assurance.
Testing is a method used within the quality assurance process, and if it is correctly managed, outsourcing some testing tasks makes some sense, especially if you don't want to invest in the resources required for a fair-sized lab. Having a contractor perform browser compatibility tests in their own lab can be a great saver of time and capital for a busy start-up venture.
Every change should be tested.
The quality control process measures all products to verify they meet a standard of quality; if you have a formal quality control process, then everything that goes out must be tested. If you outsource testing, then you must make long term decisions on how to handle all published output. You'll either need to retain a testing firm for ongoing production needs, or train your own team. Either way, the quality control process is basic to your ability to provide quality to your customers, and so should be managed by your company.
Quality assurance is about _processes_.
Quality assurance requires constant and deep involvement in the planning, design, and development phases of a major project, because QA focuses on processes. Getting QA involved up front will save you money, as studies show that bugs cost significantly less the earlier they are caught. Outsourcing QA would require a very, very close partnership with the contractors. Ask yourself if you want to include a testing firm in every design meeting -- if not, then they won't be able to act effectively in a quality assurance role.
A successful launch will fork your priorities.
When the website launches, you will find your priorities splitting in two directions: maintenance of the site and development of the next site iteration. Both phases require QA and testing attention, but your greatest resource will be attention. If you expect a concerted ongoing web effort, then you need to involve testing and QA in both tracks. Ask yourself if you want to be chained to a contractor's schedule for releasing new code. If you need to make a change to the code in response to an emergency, who tests that code before publication to the web? If testing is to be performed by a contractor, then they necessarily have a say in the scheduling by virtue of _their_ availability.
Audience identification is a critical business priority.
Quality assurance is deeply involved in the definition of the site's audience, including identification of the target browsers and platforms used by the audience. I read recently [12 May 1999] that Windows 2000 is being released OEM on some brands of PC -- this means I have to evaluate the importance of this platform to my audience, and, if my research shows it is necessary, incorporate this platform into my testing. If I outsourced my testing, I would have to wait for them to verify my site's support for this platform; if I outsourced my QA, I'd have to wait for them to decide how important this platform was to my targeted audience… but this really a business decision.
Quality assurance for web sites is a partisan role.
I feel that quality assurance is best done by people with passion towards the product, the company, and the mission. The members of a quality assurance team should be ombudsmen for their audience, and should represent the user at design and development meetings. QA staff should inject a consideration for usability into every process and discussion. I don't think you can outsource a passionate QA team, even though I'm sure you could find an outsource team passionate about doing a good job of testing. Ask yourself how important the web site is to the success of your company. If the website does more than present brouchureware, then you should consider the website a capital investment.
Who will set and maintain the standards?
When outsourcing a process or elements of a process that is by its nature ongoing, all parties involved must be clear on who sets the standards and who maintains the standards. For a testing effort, _you_ must control those standards, and _you_ must have a mechanism through which you can change those standards. You also need a mechanism through which the contractor can provide you feedback about the appropriateness and accuracy of those standards.
The management of quality assurance shouldn't be outsourced.
If you outsource any part of your testing process, you should create a full-time position in your company for somebody to manage quality assurance; this person should liaise with and set the direction for any testing contractors. A wide range of tasks fall under the umbrella of quality assurance, including testing -- usability, compatibility, integration, regression, etc. – as well as requirements management, design reviews, and reviewing customer complaints. Don't set an expectation for quality assurance when you are only paying for testing.

Advantages of Outsourcing Testing

Outsourcing some kinds of testing can have definite advantages.

getting stuff done
Even the best team runs out of the time or bodies necessary to get certain tests accomplished. If hiring a test firm to perform certain tests takes care of some test goals, then outsourcing is a great idea.
hiring expertise
Some tests are best performed by experienced professionals, either because the necessary knowledge requires specialize education or backgrounds, or because the necessary skill set can't be learned under a short deadline. Moreover, some tests require a specialized understanding in order to interpret and analyze the results, and it makes sense to outsource these tests.
hiring authority
Some kinds of tests -- and some kinds of test firm -- have an authority that a homegrown test team may not be able to compete with. Company executives often find it easier to accept critical results if they come from a third-party, rather than from their own test team (speaking from experience here).
hiring a neutral point-of-view
Sometimes having an independent team perform testing tasks can provide an objective point-of-view about quality issues like usability or compatibility. Even if I am rigorously fair in my compatibility testing, I'll still have browser prejudices; I'll still have a favorite browser and platform, and so may tend to do the majority of my testing with them. Outsourcing some testing may provide me with more data, and hopefully unbiased data.
hiring a fresh set of eyes
Involving a new team in testing exposes the code to new test tools and methods; a test firm may find defects your tools were unable to find.

Testing and Quality Assurance Firms

I don't have any direct experience with independent test firms, so this list is not an endorsement of any particular test firm. If you have suggestions, references or comments about testing firms, please drop me a line at qa@philosophe.com.

### Otivo, Inc. OTIVO serves a broad spectrum of clients - from large to small, from content to commerce to community providers, from Web sites to Web developers to communications agencies - and tests in everything from Lynx to Netscape to WebTV. What ties OTIVO's clients together is the desire to have independent validation that their Web site provides a problem-free experience for users - however they surf the Web. On their services page, Otivo offers: + Web QA test plan development and testing + QA specifications and process development + Web usability test plan development and testing

Usability Testing Firms