patterns & practices: Performance Testing Guidance for Web Applications

Updated Wiki: Home

mycodeplexuser — Thu, 28 Aug 2008 00:15:20 GMT

patterns & practices Performance Testing Guidance for Web Applications

Welcome to the patterns & practices Performance Testing Guidance for Web Applications! This guide shows you an end-to-end approach for implementing performance testing. Whether you are new to performance testing, or looking for ways to improve your current performance testing approach, you will find insights that you can tailor for your specific scenarios. This guide is related to our Performance Testing Guidance Project . - J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Forewards

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Reference

Glossary

Team

Core Team: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
Contributors and Reviewers

CREATED ISSUE: Chapter 15, datasets b and c swapped in text references

robbak11 — Wed, 09 Jan 2008 19:15:19 GMT

In chapter 15, references to datasets B and C are swapped in discussions of Medians, Normal Values and other concepts.

UPDATED WIKI: Home

sbarber — Sat, 15 Dec 2007 04:50:42 GMT

patterns & practices Performance Testing Guidance for Web Applications

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Forewards

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

UPDATED WIKI: Foreword By Rico Mariani

prashantbansode — Wed, 29 Aug 2007 20:09:34 GMT

Foreword By Rico Mariani

It’s hard to imagine anything than is considered a more arcane art than performance tuning – unless perhaps it is performance testing.
If you were to go door to door between groups just within Microsoft you would find many different approaches with various different degrees of quality or success. Pretty much everyone will vow that their approach is certainly the one that is best for them – except maybe an honest few, who might say something more modest. Some have good reason to be confident because they really have studied the space very well. In my own experience at least, the situation is not altogether different outside of Microsoft than it is inside where I do my work. It’s a mixed bag, on a good day.
If I had to describe the most common problem I see in this space with one word it would imbalance. There are many aspects to testing and teams tend to unduly focus on one or another and then sometimes get blindsided by the ones they missed. Perhaps they’re only thinking about throughput – what about consumption? Perhaps only latency – what about smooth delivery? Perhaps only cost -- what about scalability?
You get great performance by balancing the key factors, considering them in your designs and then tracking them carefully. So perhaps the greatest service that a book like Performance Testing Guidance for Web Applications can provide to you is a broader understanding of what all the factors might be so that you have an excellent menu of considerations to choose from in your testing plan. Luckily, that is just what you’re going to get.
The Guidance that follows provides a great survey of the most important considerations: From how to understand and quantify your desired end user experience, how to choose key resources for study, to advice on summarizing results in a statistically meaningful way, and how to fit these practices into different software lifecycles. And even though the focus is squarely on web applications, the teachings are actually much more general and can easily be applied for many different kinds of applications.
Great engineering comes from creating predictable results at predictable costs. In fact, I like to say that if you’re not measuring you’re not engineering. This volume will provide you with the performance testing fundamentals to give you the ongoing metrics you need to do great engineering.

Rico Mariani
Chief Architect of Visual Studio
Microsoft Corporation
July, 2007

Rico Mariani began his career at Microsoft in 1988, working on language products beginning with Microsoft® C version 6.0, and contributed there until the release of the Microsoft Visual C++® version 5.0 development system. In 1995, Rico became development manager for what was to become the "Sidewalk" project, which started his 7 years of platform work on various MSN technologies. In the summer of 2002, Rico returned to the Developer Division to as a Performance Architect on the CLR team. His performance work led to his most recent assignment as Chief Architect of Visual Studio. Rico's interests include compilers and language theory, databases, 3-D art, and good fiction.

UPDATED WIKI: Foreword By Alberto Savoia

prashantbansode — Wed, 29 Aug 2007 20:07:53 GMT

Foreword By Alberto Savoia

Testing the performance of web applications is easy. It’s easy to design unrealistic scenarios. Easy to collect and measure irrelevant performance data. And, even if you manage to design a sound scenario and collect the right data, it’s easy to use the wrong statistical methods to summarize and present the results.

Starting in the late 90s, through the peak of the Internet bubble and beyond, I spent a lot of time testing the performance of web applications. During that period, I designed and led several mission-critical web performance and load tests for high-profile Internet companies. Working with the in-house performance experts at each company was very revealing – and quite frightening. Most of the engineers assigned to work on web application performance were smart, hard-working, and dedicated; they invested in expensive software and hardware, read the right books, and followed the best practices of the day. But, somehow, the results of their performance measurements and predictions did not match reality. In some cases the performance tests overestimated the performance and scalability of the web applications – leading to embarrassing and costly crashes when the web application was deployed. In other cases, they underestimated capacity and scalability – leading to unnecessary spending on hardware and infrastructure. The errors in these tests were not small; some tests overestimated or underestimated actual performance and capacity by an order of magnitude or more! How is this possible?

Based on my experience, the majority of gross errors in web application performance testing are the result of oversimplification. More precisely, they are the result oversimplification of user behavior and oversimplification in summarizing and reporting test results. Imagine a transportation engineer estimating traffic patterns for a proposed stretch of highway by assuming that most drivers will drive at the same average speed, break and accelerate with the same response time and at the same rate, and never change lanes. A simple – but completely worthless – scenario. Or imagine the same transportation engineer reporting that there are no traffic issues because the average speed is 57mph – without bringing up that during rush-hour the average speed is 25mph. A simple, but very misleading, result. Unfortunately, most web application performance testers commit errors of oversimplification as bad, or worse, as the ones committed by our hypothetical transportation engineer.

I am all for simplicity but, as Albert Einstein once said: “Make everything as simple as possible, but not simpler.” When it comes to testing the performance of web applications, that’s exactly what this remarkable – and much needed – book teaches you. The authors leverage their passion, experience, and hard-earned knowledge and provide you with the broad, thorough, and extensible foundation you need to tackle web performance testing the right way. Performance Testing Guidance for Web Applications does not get bogged down with unnecessary details, but it does make sure that you know about – and don’t overlook – the key parameters and variables that you need to take into account in designing, conducting, and analyzing your tests.

If you are new to web performance testing, this book will get you started on the right path and save you a lot of time and embarrassment. Even if you are a seasoned web performance testing veteran, I am confident that this book will provide you with new insights and, most likely, have you slap your forehead a few times as you read about some common and familiar mistakes. In either case, Performance Testing Guidance for Web Applications, is a must-have for any web performance engineer bookshelf.

Alberto Savoia
Founder and CTO, Agitar Software Inc.
July, 2007

Author of: “The Science and Art of Web Site Load Testing”, “Web Load Test Planning”, and “Trade Secrets from a Web Testing Expert”.

UPDATED WIKI: Home

prashantbansode — Wed, 29 Aug 2007 20:04:56 GMT

patterns & practices Performance Testing Guidance for Web Applications

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Forewards

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Reference

Glossary

Team

Core Team: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
Contributors and Reviewers

Related Sites

patterns & practices Performance Testing Guidance

NEW POST: Capacity Models for Planning Resources

HamoGirl70 — Tue, 28 Aug 2007 09:56:16 GMT

How many hours of test planning does the average web shop plan for when assessing development hours to the amount of resource hours to develop test cases, scripts, execute and provide read out of the end product? For example, is there some industry average that eCare (account maint functions on the web) for companies such as utility companies use? This includes the time it takes for the resources to assess the requirements of the project.

UPDATED WIKI: Home

prashantbansode — Mon, 27 Aug 2007 20:24:33 GMT

patterns & practices Performance Testing Guidance for Web Applications

Download the Guide

The Final Release is Available! Start using the guide today, while we continue to make improvements.

Download the Performance Testing Guidance for Web Applications Guide

Parts

Part 1, Introduction to Performance Testing
Part II, Exemplar Performance Testing Approaches
Part III, Identify the Test Environment
Part IV, Identify Performance Acceptance Criteria
Part V, Plan and Design Tests
Part VI, Execute Tests
Part VII, Analyze Results and Report
Part VIII, Performance Testing Techniques

Chapters

Introduction

Part 1, Introduction to Performance Testing

Part II, Exemplar Performance Testing Approaches

Part III, Identify the Test Environment

Chapter 8 – Evaluating Systems to Increase Performance-Testing Effectiveness

Part IV, Identify Performance Acceptance Criteria

Part V, Plan and Design Tests

Part VI, Execute Tests

Chapter 14 – Test Execution

Part VII, Analyze Results and Report

Part VIII, Performance-Testing Techniques

Reference

Glossary

Team

Core Team: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
Contributors and Reviewers

Related Sites

patterns & practices Performance Testing Guidance

UPDATED RELEASE: Performance Testing Guidance - Final Release (Aug 27, 2007)

Mon, 27 Aug 2007 20:23:36 GMT

Performance Testing Guidance for Web Applications - PDF Version.
Click the link below to download the guide.

CREATED RELEASE: Performance Testing Guidance - Final Release (Aug 27, 2007)

Mon, 27 Aug 2007 20:19:42 GMT

Performance Testing Guidance for Web Applications - PDF Version.
Click the link below to download the guide.

UPDATED WIKI: Contributors and Reviewers

prashantbansode — Thu, 23 Aug 2007 19:08:24 GMT

Contributors and Reviewers

Microsoft Contributors / Reviewers

Alan Ridlehoover; Clint Huffman; Edmund Wong; Ken Perilman; Larry Brader; Mark Tomlinson; Paul Williams; Pete Coupland; Rico Mariani

External Contributors/Reviewers

Alberto Savoia; Ben Simo; Cem Kaner; Chris Loosley; Corey Goldberg; Dawn Haynes; Derek Mead; Karen N. Johnson; Mike Bonar; Pradeep Soundararajan; Richard Leeke; Roland Stens; Ross Collard; Steven Woody

UPDATED WIKI: Chapter 18 – Stress-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 19:06:51 GMT

Chapter 18 – Stress Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of stress testing.
Learn how to stress-test a Web application.

Overview

Stress testing is a type of performance testing focused on determining an application’s robustness, availability, and reliability under extreme conditions. The goal of stress testing is to identify application issues that arise or become apparent only under extreme conditions. These conditions can include heavy loads, high concurrency, or limited computational resources. Proper stress testing is useful in finding synchronization and timing bugs, interlock problems, priority problems, and resource loss bugs. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in an acceptable manner (e.g., not corrupting or losing data).

Stress tests typically involve simulating one or more key production scenarios under a variety of stressful conditions. For example, you might deploy your application on a server that is already running a processor-intensive application; in this way, your application is immediately “starved” of processor resources and must compete with the other application for processor cycles. You can also stress-test a single Web page or even a single item such as a stored procedure or class.

This chapter presents a high-level introduction to stress-testing a Web application. Stress testing can help you identify application issues that surface only under extreme conditions.

Examples of Stress Conditions

Examples of stress conditions include:

Excessive volume in terms of either users or data; examples might include a denial of service (DoS) attack or a situation where a widely viewed news item prompts a large number of users to visit a Web site during a three-minute period.
Resource reduction such as a disk drive failure.
Unexpected sequencing.
Unexpected outages/outage recovery.

Examples of Stress-Related Symptoms

Examples of stress-related symptoms include:

Data is lost or corrupted.
Resource utilization remains unacceptably high after the stress is removed.
Application components fail to respond.
Unhandled exceptions are presented to the end user.

How to Use This Chapter

Use this chapter to understand the key concepts of stress testing and the steps involved in stress-testing a Web application. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for stress-testing a Web application and the key outcomes of this type of testing.
Use the “Approach for Stress Testing” section to get** an overview of the approach for stress-testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in stress-testing a Web application.
Use the “Usage Scenario for Stress Testing” section to understand various real-world scenarios where stress testing is employed.

Input

To perform stress testing, you are likely to use as reference one or more of the following items:

Results from previous stress tests
Application usage characteristics (scenarios)
Concerns about those scenarios under extreme conditions
Workload profile characteristics
Current peak load capacity (obtained from load testing)
Hardware and network architecture and data
Disaster-risk assessment (e.g., likelihood of blackouts, earthquakes, etc.)

Output

Output from a stress test may include:

Measures of the application under stressful conditions
Symptoms of the application under stress
Information the team can use to address robustness, availability, and reliability

Approach for Stress Testing

The following steps are involved in stress-testing a Web application:

Step1 - Identify test objectives. Identify the objectives of stress testing in terms of the desired outcomes of the testing activity.
Step 2 - Identify key scenario(s). Identify the application scenario or cases that need to be stress-tested to identify potential problems.
Step 3 - Identify the workload. Identify the workload that you want to apply to the scenarios identified during the “Identify objectives” step. This is based on the workload and peak load capacity inputs.
Step 4 - Identify metrics. Identify the metrics that you want to collect about the application’s performance. Base these metrics on the potential problems identified for the scenarios you identified during the “Identify objectives” step.
Step 5 - Create test cases. Create the test cases in which you define steps for running a single test, as well as your expected results.
Step 6 - Simulate load. Use test tools to simulate the required load for each test case and capture the metric data results.
Step 7 - Analyze results. Analyze the metric data captured during the test.

These steps are graphically represented below; the following sections discuss each step in detail.

_Figure 18.1 Stress Testing Steps_

Step 1 - Identify Test Objectives

Asking yourself or others the following questions can help in identifying the desired outcomes of your stress testing:

Is the purpose of the test to identify the ways the system can possibly fail catastrophically in production?
Is it to provide information to the team in order to build defenses against catastrophic failures?
Is it to identify how the application behaves when system resources such as memory, disk space, network bandwidth, or processor cycles are depleted?
Is it to ensure that functionality does not break under stress? For example, there may be cases where operational performance metrics meet the objectives, but the functionality of the application is failing to meet them — orders are not inserted in the database, the application is not returning the complete product information in searches, form controls are not being populated properly, redirects to custom error pages are occurring during the stress testing, and so on.

Step 2 - Identify Key Scenario(s)

To get the most value out of a stress test, the test needs to focus on the behavior of the usage scenario or scenarios that matter most to the overall success of the application. To identify these scenarios, you generally start by defining a single scenario that you want to stress-test in order to identify a potential performance issue. Consider these guidelines when choosing appropriate scenarios:

Select scenarios based on how critical they are to overall application performance.
Try to test those operations that are most likely to affect performance. These might include operations that perform intensive locking and synchronization, long transactions, and disk-intensive input/output (I/O) operations.
Base your scenario selection on the specific areas of your application identified as potential bottlenecks by load-testing data. Although you should have fine-tuned and removed the bottlenecks after load testing, you should still stress-test the system in these areas to verify how well your changes handle extreme stress levels.

Examples of scenarios that may need to be stress tested separately from other usage scenarios for a typical e-commerce application include the following:

An order-processing scenario that updates the inventory for a particular product. This functionality has the potential to exhibit locking and synchronization problems.
A scenario that pages through search results based on user queries. If a user specifies a particularly wide query, there could be a large impact on memory utilization. For example, memory utilization could be affected if a query returns an entire data table.

Step 3 - Identify the Workload

The load you apply to a particular scenario should stress the system sufficiently beyond threshold limits to enable you to observe the consequences of the stress condition. One method to determine the load at which an application begins to exhibit signs of stress is to incrementally increase the load and observe the application behavior under various load conditions. The key is to systematically test with various workloads until you create a significant failure. These variations may be accomplished by adding more users, reducing delay times, adding or reducing the number and type of user activities represented, or adjusting test data.

For example, a stress test could be designed to simulate every registered user of the application attempting to log on during one 30-second period. This would simulate a situation where the application suddenly became available again after a period of downtime and all users were anxiously refreshing their browsers, waiting for the application to come back online. Although this situation does not occur frequently in the real world, it does happen often enough for there to be real value in learning how the application will respond if it does.

Remember to represent the workload with accurate and realistic test data — type and volume, different user logins, product IDs, product categories, and so on — allowing you to simulate important failures such as deadlocks or resource consumption.

The following activities are generally useful in identifying appropriate workloads for stress testing:

Identify the distribution of work. For each key scenario, identify the distribution of work to be simulated. The distribution is based on the number and type of users executing the scenario during the stress test.
Estimate peak user loads. Identify the maximum expected number of users during peak load conditions for the application. Using the work distribution you identified for each scenario, calculate the percentage of user load per key scenario.
Identify the anti-profile. As an alternative, you can start by applying an anti-profile to the normal workload. In an anti-profile, the workload distributions are inverted for the scenario under consideration. For example, if the normal load for the order-processing scenario is 10 percent of the total workload, the anti-profile would be 90 percent of the total workload. The remaining load can be distributed among the other scenarios. Using an anti-profile can serve as a valuable starting point for your stress tests because it ensures that the critical scenarios are subjected to loads beyond the normal load conditions.

Step 4 - Identify Metrics

When identified and captured correctly, metrics provide information about how well or poorly your application is performing as compared to your performance objectives. In addition, metrics can help you identify problem areas and bottlenecks within your application.

Using the desired performance characteristics identified during the “Identify objectives” step, identify metrics to be captured that focus on potential pitfalls for each scenario. The metrics can be related to both performance and throughput goals as well as providing information about potential problems; for example, custom performance counters that have been embedded in the application.

When identifying metrics, you will use either direct objectives or indicators that are directly or indirectly related to those objectives. The following table describes performance metrics in terms of related performance objectives.

Performance metrics	Category
Base set of metrics
Processor	Processor utilization
Process	Memory consumption
	Processor utilization
	Process recycles
Memory	Memory available
	Memory utilization
Disk	Disk utilization
Network	Network utilization
Transactions/business metrics	Transactions/sec
	Transactions succeeded
	Transactions failed
	Orders succeeded
	Orders failed
Threading	Contentions per second
	Deadlocks
	Thread allocation
Response times	Transactions times

Step 5 - Create Test Cases

Identifying workload profiles and key scenarios generally does not provide all of the information necessary to implement and execute test cases. Additional inputs for completely designing a stress test include performance objectives, workload characteristics, test data, test environments, and identified metrics. Each test design should mention the expected results and/or the key data of interest to be collected, in such a way that each test case can be marked as a “pass,” “fail,” or “inconclusive” after execution.

The following is an example of a test case based on the order-placement scenario.

Test 1 – Place Order Scenario

Workload: 1,000 simultaneous users.
Think time: Use a random think time between 1 and 10 seconds in the test script after each operation.
Test Duration: Run the test for two days.

Expected results:

Application hosting process should not recycle because of deadlock or memory consumption.
Throughput should not fall below 35 requests per second.
Response time should not be greater than 7 seconds for 95 percent of total transactions completed.
“Server busy” errors should not be more than 10 percent of the total response because of contention-related issues.
Order transactions should not fail during test execution. Database entries should match the “Transactions succeeded” count.

Step 6 - Simulate Load

After you have completed the previous steps to an appropriate degree, you should be ready to simulate the load executing the stress test. Typically, test execution follows these steps:

Validate that the test environment matches the configuration that you were expecting and/or designed your test for.
Ensure that both the test and the test environment are correctly configured for metrics collection.
Before running the test, execute a quick “smoke test” to make sure that the test script and remote performance counters are working correctly.
Reset the system (unless your scenario is to do otherwise) and start a formal test execution.

Note: Make sure that the client (a.k.a. load generator) computers that you use to generate load are not overly stressed. Utilization of resources such as processor and memory should remain low enough to ensure that the load-generation environment is not itself a bottleneck.

Step 7 - Analyze Results

Analyze the captured data and compare the results against the metric’s accepted level. If the results indicate that your required performance levels have not been attained, analyze and fix the cause of the bottleneck. To address observed issues, you might need to do one or more of the following:

Perform a design review.
Perform a code review.
Run stress tests in environments where it is possible to debug possible causes of failures, during test execution.

In situations where performance issues are observed, but only under conditions that are deemed to be unlikely enough to warrant tuning at the current time, you may want to consider conducting additional tests to identify an early indicator for the issue in order to avoid unwanted surprises.

Usage Scenarios for Stress Testing

The following are examples of how stress testing is applied in practice:

Application stress testing. This type of test typically focuses on more than one transaction on the system under stress, without the isolation of components. With application stress testing, you are likely to uncover defects related to data locking and blocking, network congestion, and performance bottlenecks on different components or methods across the entire application. Because the test scope is a single application, it is common to use this type of stress testing after a robust application load-testing effort, or as a last test phase for capacity planning. It is also common to find defects related to race conditions and general memory leaks from shared code or components.
Transactional stress testing. Transactional stress tests aim at working at a transactional level with load volumes that go beyond those of the anticipated production operations. These tests are focused on validating behavior under stressful conditions, such as high load with same resource constraints, when testing the entire application. Because the test isolates an individual transaction, or group of transactions, it allows for a very specific understanding of throughput capacities and other characteristics for individual components without the added complication of inter-component interactions that occurs in testing at the application level. These tests are useful for tuning, optimizing, and finding error conditions at the specific component level.
*Systemic stress testing.* In this type of test, stress or extreme load conditions are generated across multiple applications running on the same system, thereby pushing the boundaries of the applications’ expected capabilities to an extreme. The goal of systemic stress testing is to uncover defects in situations where different applications block one another and compete for system resources such as memory, processor cycles, disk space, and network bandwidth. This type of testing is also known as integration stress testing or consolidation stress testing. In large-scale systemic stress tests, you stress all of the applications together in the same consolidated environment. Some organizations choose to perform this type of testing in a larger test lab facility, sometimes with the hardware or software vendor’s assistance.

Exploratory Stress Testing

Exploratory stress testing is an approach to subjecting a system, application, or component to a set of unusual parameters or conditions that are unlikely to occur in the real world but are nevertheless possible. In general, exploratory testing can be viewed as an interactive process of simultaneous learning, test design, and test execution. Most often, exploratory stress tests are designed by modifying existing tests and/or working with application/system administrators to create unlikely but possible conditions in the system. This type of stress testing is seldom conducted in isolation because it is typically conducted to determine if more systematic stress testing is called for related to a particular failure mode. The following are some examples of exploratory stress tests to determine the answer to “How will the system respond if…?”

All of the users logged on at the same time.
The load balancer suddenly failed.
All of the servers started their scheduled virus scan at the same time during a period of peak load.
The database went offline during peak usage.

Summary

Stress testing allows you to identify potential application issues that surface only under extreme conditions. Such conditions range from exhaustion of system resources such as memory, processor cycles, network bandwidth, and disk capacity to excessive load due to unpredictable usage patterns, common in Web applications.

Stress testing centers around objectives and key user scenarios with an emphasis on the robustness, reliability, and stability of the application. The effectiveness of stress testing relies on applying the correct methodology and being able to effectively analyze testing results. Applying the correct methodology is dependent on the capacity for reproducing workload conditions for both user load and volume of data, reproducing key scenarios, and interpreting the key performance metrics.

UPDATED WIKI: Chapter 18 – Stress-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 19:06:15 GMT

Chapter 18 – Stress Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of stress testing.
Learn how to stress-test a Web application.

Overview

Examples of Stress Conditions

Examples of stress conditions include:

Excessive volume in terms of either users or data; examples might include a denial of service (DoS) attack or a situation where a widely viewed news item prompts a large number of users to visit a Web site during a three-minute period.
Resource reduction such as a disk drive failure.
Unexpected sequencing.
Unexpected outages/outage recovery.

Examples of Stress-Related Symptoms

Examples of stress-related symptoms include:

Data is lost or corrupted.
Resource utilization remains unacceptably high after the stress is removed.
Application components fail to respond.
Unhandled exceptions are presented to the end user.

How to Use This Chapter

Use this chapter to understand the key concepts of stress testing and the steps involved in stress-testing a Web application. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for stress-testing a Web application and the key outcomes of this type of testing.
Use the “Approach for Stress Testing” section to get** an overview of the approach for stress-testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in stress-testing a Web application.
Use the “Usage Scenario for Stress Testing” section to understand various real-world scenarios where stress testing is employed.

Input

To perform stress testing, you are likely to use as reference one or more of the following items:

Results from previous stress tests
Application usage characteristics (scenarios)
Concerns about those scenarios under extreme conditions
Workload profile characteristics
Current peak load capacity (obtained from load testing)
Hardware and network architecture and data
Disaster-risk assessment (e.g., likelihood of blackouts, earthquakes, etc.)

Output

Output from a stress test may include:

Measures of the application under stressful conditions
Symptoms of the application under stress
Information the team can use to address robustness, availability, and reliability

Approach for Stress Testing

The following steps are involved in stress-testing a Web application:

Step1 - Identify test objectives. Identify the objectives of stress testing in terms of the desired outcomes of the testing activity.
Step 2 - Identify key scenario(s). Identify the application scenario or cases that need to be stress-tested to identify potential problems.
Step 3 - Identify the workload. Identify the workload that you want to apply to the scenarios identified during the “Identify objectives” step. This is based on the workload and peak load capacity inputs.
Step 4 - Identify metrics. Identify the metrics that you want to collect about the application’s performance. Base these metrics on the potential problems identified for the scenarios you identified during the “Identify objectives” step.
Step 5 - Create test cases. Create the test cases in which you define steps for running a single test, as well as your expected results.
Step 6 - Simulate load. Use test tools to simulate the required load for each test case and capture the metric data results.
Step 7 - Analyze results. Analyze the metric data captured during the test.

These steps are graphically represented below; the following sections discuss each step in detail.

Cannot resolve link: [image:Fig18.1.GIF]

_Figure 18.1 Stress Testing Steps_

Step 1 - Identify Test Objectives

Asking yourself or others the following questions can help in identifying the desired outcomes of your stress testing:

Is the purpose of the test to identify the ways the system can possibly fail catastrophically in production?
Is it to provide information to the team in order to build defenses against catastrophic failures?
Is it to identify how the application behaves when system resources such as memory, disk space, network bandwidth, or processor cycles are depleted?
Is it to ensure that functionality does not break under stress? For example, there may be cases where operational performance metrics meet the objectives, but the functionality of the application is failing to meet them — orders are not inserted in the database, the application is not returning the complete product information in searches, form controls are not being populated properly, redirects to custom error pages are occurring during the stress testing, and so on.

Step 2 - Identify Key Scenario(s)

Select scenarios based on how critical they are to overall application performance.
Try to test those operations that are most likely to affect performance. These might include operations that perform intensive locking and synchronization, long transactions, and disk-intensive input/output (I/O) operations.
Base your scenario selection on the specific areas of your application identified as potential bottlenecks by load-testing data. Although you should have fine-tuned and removed the bottlenecks after load testing, you should still stress-test the system in these areas to verify how well your changes handle extreme stress levels.

Examples of scenarios that may need to be stress tested separately from other usage scenarios for a typical e-commerce application include the following:

An order-processing scenario that updates the inventory for a particular product. This functionality has the potential to exhibit locking and synchronization problems.
A scenario that pages through search results based on user queries. If a user specifies a particularly wide query, there could be a large impact on memory utilization. For example, memory utilization could be affected if a query returns an entire data table.

Step 3 - Identify the Workload

Identify the distribution of work. For each key scenario, identify the distribution of work to be simulated. The distribution is based on the number and type of users executing the scenario during the stress test.
Estimate peak user loads. Identify the maximum expected number of users during peak load conditions for the application. Using the work distribution you identified for each scenario, calculate the percentage of user load per key scenario.
Identify the anti-profile. As an alternative, you can start by applying an anti-profile to the normal workload. In an anti-profile, the workload distributions are inverted for the scenario under consideration. For example, if the normal load for the order-processing scenario is 10 percent of the total workload, the anti-profile would be 90 percent of the total workload. The remaining load can be distributed among the other scenarios. Using an anti-profile can serve as a valuable starting point for your stress tests because it ensures that the critical scenarios are subjected to loads beyond the normal load conditions.

Step 4 - Identify Metrics

Performance metrics	Category
Base set of metrics
Processor	Processor utilization
Process	Memory consumption
	Processor utilization
	Process recycles
Memory	Memory available
	Memory utilization
Disk	Disk utilization
Network	Network utilization
Transactions/business metrics	Transactions/sec
	Transactions succeeded
	Transactions failed
	Orders succeeded
	Orders failed
Threading	Contentions per second
	Deadlocks
	Thread allocation
Response times	Transactions times

Step 5 - Create Test Cases

Test 1 – Place Order Scenario

Application hosting process should not recycle because of deadlock or memory consumption.
Throughput should not fall below 35 requests per second.
Response time should not be greater than 7 seconds for 95 percent of total transactions completed.
“Server busy” errors should not be more than 10 percent of the total response because of contention-related issues.
Order transactions should not fail during test execution. Database entries should match the “Transactions succeeded” count.

Step 6 - Simulate Load

After you have completed the previous steps to an appropriate degree, you should be ready to simulate the load executing the stress test. Typically, test execution follows these steps:

Validate that the test environment matches the configuration that you were expecting and/or designed your test for.
Ensure that both the test and the test environment are correctly configured for metrics collection.
Before running the test, execute a quick “smoke test” to make sure that the test script and remote performance counters are working correctly.
Reset the system (unless your scenario is to do otherwise) and start a formal test execution.

Step 7 - Analyze Results

Perform a design review.
Perform a code review.
Run stress tests in environments where it is possible to debug possible causes of failures, during test execution.

Usage Scenarios for Stress Testing

The following are examples of how stress testing is applied in practice:

Application stress testing. This type of test typically focuses on more than one transaction on the system under stress, without the isolation of components. With application stress testing, you are likely to uncover defects related to data locking and blocking, network congestion, and performance bottlenecks on different components or methods across the entire application. Because the test scope is a single application, it is common to use this type of stress testing after a robust application load-testing effort, or as a last test phase for capacity planning. It is also common to find defects related to race conditions and general memory leaks from shared code or components.
Transactional stress testing. Transactional stress tests aim at working at a transactional level with load volumes that go beyond those of the anticipated production operations. These tests are focused on validating behavior under stressful conditions, such as high load with same resource constraints, when testing the entire application. Because the test isolates an individual transaction, or group of transactions, it allows for a very specific understanding of throughput capacities and other characteristics for individual components without the added complication of inter-component interactions that occurs in testing at the application level. These tests are useful for tuning, optimizing, and finding error conditions at the specific component level.
*Systemic stress testing.* In this type of test, stress or extreme load conditions are generated across multiple applications running on the same system, thereby pushing the boundaries of the applications’ expected capabilities to an extreme. The goal of systemic stress testing is to uncover defects in situations where different applications block one another and compete for system resources such as memory, processor cycles, disk space, and network bandwidth. This type of testing is also known as integration stress testing or consolidation stress testing. In large-scale systemic stress tests, you stress all of the applications together in the same consolidated environment. Some organizations choose to perform this type of testing in a larger test lab facility, sometimes with the hardware or software vendor’s assistance.

Exploratory Stress Testing

All of the users logged on at the same time.
The load balancer suddenly failed.
All of the servers started their scheduled virus scan at the same time during a period of peak load.
The database went offline during peak usage.

Summary

UPDATED WIKI: Chapter 17 – Load-Testing Web Applications

prashantbansode — Thu, 23 Aug 2007 18:49:14 GMT

Chapter 17 – Load-Testing Web Applications

- J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea

Objectives

Understand the key concepts of load testing.
Learn how to load-test a Web application.

Overview

This chapter explains how to load-test a Web application. Load testing helps to identify the maximum operating capacity of an application as well as any bottlenecks that might interfere with its operating at capacity. The basic approach to performing load testing on a Web application is:

Identify the performance-critical scenarios.
Identify the workload profile for distributing the entire load among the key scenarios.
Identify the metrics that you want to collect in order to verify them against your performance objectives.
Design tests to simulate the load.
Use tools to implement the load according to the designed tests, and capture the metrics.
Analyze the metric data captured during the tests.

By using an iterative testing process, these steps should help you achieve your performance objectives.

There are many reasons for load-testing a Web application. The most basic type of load testing is used to determine the Web application’s behavior under both normal and anticipated peak load conditions. As you begin load testing, it is recommended that you start with a small number of virtual users and then incrementally increase the load from normal to peak. You can then observe how your application performs during this gradually increasing load condition. Eventually, you will cross a threshold limit for your performance objectives. For example, you might continue to increase the load until the server processor utilization reaches 75 percent, or when end-user response times exceed 8 seconds.

How to Use This Chapter

Use this chapter to understand the key concepts of load testing and the steps involved in load-testing Web applications. To get the most from this chapter:

Use the “Input” and “Output” sections to understand the key inputs for load testing a Web application and the key outcomes of doing so.
Use the “Approach for Load Testing” section to get** an overview of the approach for load testing a Web application, and as quick reference guide for you and your team.
Use the various steps sections to understand the details of each step involved in load-testing a Web application.

Input

The following are useful inputs for load-testing a Web application:

Performance-critical usage scenarios
Workload models
Performance acceptance criteria
Performance metrics associated with the acceptance criteria
Interview feedback from the designer or developer of the Web application
Interview feedback from end users of the application
Interview feedback from the operations personnel who will maintain and manage the application

Output

The main outcomes that load testing helps you to accomplish are:

Updated test plans and test designs for load and performance testing
Various performance measures such as throughput, response time, and resource utilization
Potential bottlenecks that need to be analyzed in the white-box testing phase
The behavior of the application at various load levels

Approach for Load Testing

The following steps are involved in load-testing a Web application:

Step 1 - Identify performance acceptance criteria
Step 2 - Identify key scenarios
Step 3 - Create a workload model
Step 4 - Identify the target load levels
Step 5 - Identify metrics
Step 6 - Design specific tests
Step 7 - Run tests
Step 8 - Analyze the results

These steps are graphically represented below. The sections that follow discuss each step in detail.

_Figure 17.1 Load Testing Steps_

Step 1 - Identify Performance Acceptance Criteria

Identifying performance acceptance criteria is most valuable when initiated early in the application’s development life cycle. It is frequently valuable to record the acceptance criteria for your application and store them in a place and format that is available to the entire team for review and comment. Criteria are typically determined by balancing your business, industry, technology, competitive, and user requirements.

Test objectives frequently include the following:

Response time. For example, the product catalog must be displayed in less than 3 seconds.
Throughput. For example, the system must support 100 transactions per second.
Resource utilization. A frequently overlooked aspect is the amount of resources your application is consuming, in terms of processor, memory, disk input output (I/O), and network I/O.
Maximum user load. This test objective determines how many users can run on a specific hardware configuration.
Business related metrics. This objective is mapped to business volume at normal and peak values; for example, the number of orders or Help desk calls handled at a given time.

Step 2 - Identify Key Scenarios

Scenarios are anticipated user paths that generally incorporate multiple application activities. Key scenarios are those for which you have specific performance goals, those considered to be high-risk, those that are most commonly used, or those with a significant performance impact. The basic steps for identifying key scenarios are.

Identify all the scenarios for a Web application. For example, even the most basic e-commerce application must support the following user scenarios:

Browse catalog
Search for a product
Place an order

Identify the activities involved in each of the scenarios. For example, a “Place an Order” scenario will include the following activities:

Log on to the application.
Browse the product catalog.
Search for a specific product.
Add items to the shopping cart.
Validate credit card details and place an order.

Identify the scenarios that are most commonly executed or most resource-intensive; these will be the key scenarios used for load testing. For example, in an e-commerce application, browsing a catalog may be the most commonly executed scenario, whereas placing an order may be the most resource-intensive scenario because it accesses the database.

The most commonly executed scenarios for an existing Web application can be determined by examining the log files.
The most commonly executed scenarios for a new Web application can be obtained from market research, historical data, market trends, and so on.
Resource-intensive scenarios can be identified by using design documents or the actual code implementation. The primary resources are:

Processor
Memory
Disk I/O
Network I/O

Once they have been identified, you will use these key scenarios to create workload profiles and to design load tests.

Step 3 - Create a Workload Model

When defining workload distribution, consider the following key points for determining the characteristics for user scenarios:

A user scenario is defined as a navigational path, including intermediate steps or activities, taken by the user to complete a task. This can also be thought of as a user session.
A user will typically pause between pages during a session. This is known as user delay or think time.
A session will have an average duration when viewed across multiple users. It is important to account for this when defining the load levels that will translate into concurrent usage, overlapping users, or user sessions per unit of time.
Not all scenarios can be performed by a new user, a returning user, or either; know who you expect your primary users to be and test accordingly.

Step 4 - Identify Target Load Levels

Identify the load levels to be applied to the workload distribution(s) identified during the previous step. The purpose of identifying target load levels is to ensure that your tests can be used to predict or compare a variety of production load conditions. The following are common inputs used for determining target load levels:

Business volume (both current and projected) as it relates to your performance objectives
Key scenarios
Distribution of work
Session characteristics (navigational path, duration, percentage of new users)

By combining the items above, you can determine the remaining details necessary to implement the workload model under a particular target load.

Step 5 - Identify Metrics

There is a virtually unlimited number of metrics that can be collected during a performance test execution. However, collecting too many metrics can make analysis unwieldy as well as negatively impact the application’s actual performance. For these reasons, it is important to identify the metrics that are most relevant to your performance objectives and those that you anticipate will help you to identify bottlenecks. Only well-selected metrics that are analyzed correctly and contextually provide information of value.

The following are a few suggestions for identifying the metrics that will provide the most valuable information to your project:

Define questions related to your application performance that can be easily tested. For example, what is the checkout response time when placing an order? How many orders are placed in a minute? These questions have definite answers.
With the answers to these questions, determine quality goals for comparison against external expectations. For example, checkout response time should be 30 seconds, and a maximum of 10 orders should be placed in a minute. The answers are based on market research, historical data, market trends, and so on.
Identify the metrics. Using your list of performance-related questions and answers, identify the metrics that provide information related to those questions and answers.
Identify supporting metrics. Using the same approach, you can identify lower-level metrics that focus on measuring the performance and identifying the bottlenecks in the system. When identifying low-level metrics, most teams find it valuable to determine a baseline for those metrics under single-user and/or normal load conditions. This helps you determine the acceptable load levels for your application. Baseline values help you analyze your application performance at varying load levels and serve as a starting point for trend analysis across builds or releases.
Reevaluate the metrics to be collected regularly. Goals, priorities, risks, and current issues are bound to change over the course of a project. With each of these changes, different metrics may provide more value than the ones that have previously been identified.

Additionally, to evaluate the performance of your application in more detail and to identify potential bottlenecks, it is frequently useful to monitor metrics in the following categories:

Network-specific metrics. This set of metrics provides information about the overall health and efficiency of your network, including routers, switches, and gateways.
System-related metrics. This set of metrics helps you identify the resource utilization on your server. The resources being utilized are processor, memory, disk I/O, and network I/O.
Platform-specific metrics. Platform-specific metrics are related to software that is used to host your application, such as the Microsoft .NET Framework common language runtime (CLR) and ASP.NET-related metrics.
Application-specific metrics. These include custom performance counters inserted in your application code to monitor application health and identify performance issues. You might use custom counters to determine the number of concurrent threads waiting to acquire a particular lock, or the number of requests queued to make an outbound call to a Web service.
Service-level metrics. These metrics can help to measure overall application throughput and latency, or they might be tied to specific business scenarios.
Business metrics. These metrics are indicators of business-related information, such as the number of orders placed in a given timeframe.

Step 6 - Design Specific Tests

Using your scenarios, key metrics, and workload analysis, you can now design specific tests to be conducted. Each test will generally have a different purpose, collect different data, include different scenarios, and have different target load levels. The key is to design tests that will help the team collect the information it needs in order to understand, evaluate, or tune the application.

Points to consider when designing tests include:

Do not change your test design because the design is difficult to implement in your tool.
If you cannot implement your test as designed, ensure that you record the details pertaining to the test that you do implement.
Ensure that the model contains all of the supplementary data needed to create the actual test.
Consider including invalid data in your performance tests. For example, include some users who mistype their password on the first attempt but get it correct on a second try.
First-time users usually spend significantly more time on each page or activity than experienced users.
The best possible test data is test data collected from a production database or log file.
Think about nonhuman system users and batch processes as well as end users. For example, there might be a batch process that runs to update the status of orders while users are performing activities on the site. In this situation, you would need to account for those processes because they might be consuming resources.
Do not get overly caught up in striving for perfection, and do not fall into the trap of oversimplification. In general, it is a good idea to start executing tests when you have a reasonable test designed and then enhance the design incrementally while collecting results.

Step 7 - Run Tests

Poor load simulations can render all of the work in the previous activities useless. To understand the data collected from a test execution, the load simulation must reflect the test design. When the simulation does not reflect the test design, the results are prone to misinterpretation. Consider the following steps when preparing to simulate load:

Configure the test environment in such a way that it mirrors your production environment as closely as possible, noting and accounting for all differences between the two.
Ensure that performance counters relevant for identified metrics and resource utilization are being measured and are not interfering with the accuracy of the simulation.
Use appropriate load-generation tools to create a load with the characteristics specified in your test design.
Using the load-generation tool(s), execute tests by first building up to the target load specified in your test design, in order to validate the correctness of the simulation. Some things to consider during test execution include:

Begin load testing with a small number of users distributed against your user profile, and then incrementally increase the load. It is important to allow time for the system to stabilize between increases in load while evaluating the correctness of the simulation.
Consider continuing to increase the load and record the behavior until you reach the threshold for the resources identified in your performance objectives, even if that load is beyond the target load specified in the test design. Information about when the system crosses identified thresholds is just as important as the value of the metrics at the target load of the test.
Similarly, it is frequently valuable to continue to increase the number of users until you run up against the service-level limits beyond which you would be violating your SLAs for throughput, response time, and resource utilization.

Note: Make sure that the client computers (agents) you use to generate load are not overly stressed. Resource utilization such as processor and memory must remain well below the utilization threshold values to ensure accurate test results.

Step 8 - Analyze the Results

You can analyze the test results to find performance bottlenecks between each test run or after all testing has been completed. Analyzing the results correctly requires training and experience with graphing correlated response time and system data.

The following are the steps for analyzing the data:

Analyze the captured data and compare the results against the metric’s accepted level to determine whether the performance of the application being tested shows a trend toward or away from the performance objectives.
Analyze the measured metrics to diagnose potential bottlenecks. Based on the analysis, if required, capture additional metrics in subsequent test cycles. For example, suppose that during the first iteration of load tests, the process shows a marked increase in memory consumption, indicating a possible memory leak. In the subsequent iterations, additional memory counters related to generations can be captured to study the memory allocation pattern for the application.

Summary

Load testing helps to identify the maximum operating capacity of the application and any bottlenecks that might be degrading performance.

The basic methodology for performing load testing on a Web application is to identify the performance-critical key scenarios; identify the workload profile for distributing all the load among the key scenarios; identify metrics that you want to collect in order to verify them against your performance objectives; create test cases that will be used to simulate the load test; use tools to simulate the load according to the test cases and capture the metrics; and finally, analyze the metrics data captured during the tests.