Performance Testing has become a do-or-die factor for products in the market but it is often been neglected. The industry is now more serious about doing performance testing than it was 10 years back. This article is about basic testing and performance testing concepts to help you start and get an overview of performance testing.
What is Testing?
It is the way or process used to help identify the correctness, completeness, security, and quality of product. It is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding errors. Quality is not an absolute; it is value to some person. With that in mind, testing can never completely establish the correctness of product. It furnishes a criticism or comparison that compares the state and behavior of the product against a specification.
Software testing is divided into two categories:
- Functional Testing
- Non Functional Testing
Functional Testing
Functional testing is testing the application against business requirements. It is done using the functional specifications provided by the client or by using the design specifications like use cases provided by the design team. It is done to check whether the application functionality is as per the business requirements. For eg. In business requirements document, it is mentioned that telephone number field should accept only digits. In functional testing we will check with all the values (positive/negative) whether it passes the condition as mentioned in requirement document.
Functional Testing covers:
- Unit Testing
- Smoke testing / Sanity testing
- Integration Testing (Top Down, Bottom up Testing)
- Interface & Usability Testing
- System Testing
- Regression Testing
- Pre User Acceptance Testing (Alpha & Beta)
- User Acceptance Testing
- White Box & Black Box Testing
- Globalization & Localization Testing
Non Functional Testing
In contrast to functional testing, which establishes the correct operation of the software (correct in that it matches the expected behavior defined in the design requirements), non-functional testing verifies that the software functions properly even when it receives invalid or unexpected inputs. Software fault injection, in the form of fuzzing, is an example of non-functional testing.
The non functional aspect of software components involves things or criteria which judges the operation of the system, rather than specific behavior.
Non-Functional Testing covers:
- Load Testing
- Endurance/Soak Testing
- Stress Testing
- Volume Testing
- Compatibility & Migration Testing
- Data Conversion Testing
- Security / Penetration Testing
- Scalability Testing
What is Performance Testing?
Performance testing is type of Non functional testing. It is done to determine the scalability, reliability and availability of system (hardware, software, network). This process involves different performance tests done in a dedicated testing environment which should be a replica of the production environment measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions.
Performance testing serves different purposes:
- It can demonstrate that the system meets performance criteria
- It can compare two systems to find which performs bett
- Or it can measure what parts of the system or workload causes the system to perform badl
Why Do Performance Testing?
The answer in simple words is “If you want to outperform the competitors, do it, else leave it!!”.
Performance testing is neglected in many organizations but one’s who do it are way ahead. Performance testing is mostly conducted to address risks related to operating expense, opportunity costs, continuity, and/or corporate reputation.
- Analyzing the behavior of the application at various load levels
- Identifying bottlenecks in the application/system
- Determining the performance before and after changes to the software
- Evaluating the adequacy of current capacity
- Determining the capacity of the application’s infrastructure, as well as determining the future resources required to deliver acceptable application performance
- Providing comparisons between the application’s current and desired performance characteristics
- Comparing different system configurations to determine which works best for both the application and the business
- Providing information related to the speed, scalability, and stability of a product prior to production release
- Providing data indicating the likelihood of user dissatisfaction with the performance characteristics of the system
What is Load Testing?
Load Test is usually conducted to understand the behavior of the application under a specific expected load.
The load can be the expected simultaneous or concurrent number of users on the application/system performing a specific number of transactions within the set duration. This test will give out the response times of all the important business critical transactions. Monitoring should be enabled on app servers, database servers, network, etc, to find any possible bottlenecks in the application/system which hampers the performance.
What is Stress Testing?
Stress Testing is done to understand the upper limits of capacity within the application landscape. It is performed to determine if the application will perform sufficiently if the current load goes well above the expected maximum. There are many ways to stress the system/application. System/application can be stressed by removing the think time between the transactions, pacing between iterations, reducing the hardware configuration of the servers and many more other ways.
What is Endurance Testing (Soak Testing)?
Endurance Testing is performed to determine if the application can sustain the continuous expected load. During endurance tests, memory utilization is monitored to detect potential leaks. Also important, but often overlooked is performance degradation. That is, to ensure that the throughput and/or response times after some long period of sustained activity are as good as or better than at the beginning of the test. It is recommended to run endurance tests for 12/18/24/48 hours duration depending on requirements or business needs.
What is Spike Testing?
Spike testing, as the name suggests is done by spiking (sudden increase) the number of users and understanding the behavior of the application; whether performance will suffer, the application will fail, or will it be able to handle dramatic changes in load. Example for this kind of scenario is online shopping websites, which face unexpected number of users in a very short duration during any festive offer season. If spike testing is not done, their servers may crash and you better know whats happens when servers crash.
What is Configuration Testing?
Configuration testing is another variation on traditional performance testing. Rather than testing for performance from the perspective of load, configuration testing is done to determine the effects of configuration changes in the application landscape on application performance and behavior. A common example would be experimenting with different methods of load-balancing or different network switching methodologies.
What is Isolation Testing?
Isolation testing is not unique to performance testing but a term used to describe repeating a test execution that resulted in an application problem. Often used to isolate and confirm the fault domain.
Performance of application should be taken care of or kept in mind from the beginning of the development life cycle. If you one of the rarest lucky person, performance issues can be solved after deployment in production environment. In worst case the situation can be “Start from the scratch, nothing can make it perform now”, and surely no one would like to be in this situation. The cost of developing a product is very high. No company would want/expect to have a product which don’t perform in the real world. The first thing that happens in this situation is “You are fired !!”. Performance Testing Concepts should be clear to judge and provide the solution for the performance issues in the system/application.
Performance Test Plan
Performance Test Plan is a detailed plan that specifies how performance testing will proceed from a business and technical perspective. It describes the timeline for all testing efforts, including what types of tests will be performed (eg., load tests, stress tests, endurance tests, batch performance tests, etc.). Performance Test Plan should also indicate how the performance testing team will interact with the development, deployment and support groups within the enterprise. It should also contain the names of the product owner, technical architect, developers and others who are part of the project.
A typical Performance Test Plan must contain the following points:
- Objective of the Project
- Objective of Performance Testing
- Project Contacts and Approvals
- System Architecture
- Business Processes for Performance Testing
- Response time and Volume SLAs
- Performance testing activities
- In-scope business processes
- Out-of-scope business processes
- Performance testing scenarios
- Performance test execution
- Performance test metrics
- Assumptions
- Risks
- High Level Testing Schedule
The following checklist can be used when defining the steps of a comprehensive performance testing process:
- List all the deliverables for every phase of the project, including the pilot, code freeze and production phases.
- Create or ask for the physical and logical architecture diagrams that shows every single detail.
- All the hardware and software requirements should be listed.
- Determine the database volume requirements.
- Define all performance testing objectives.
- Take into consideration the performance benchmarks (if available).
- Identify the total expected user load.
- Mention the number of simultaneous and concurrent users.
- Determine the user load mix.
- Define the response time SLA for every transaction.
- Identify the number of performance tests to be executed during the testing cycle.
- Identify and note down the schedule of any batch process that will or will not run when performance tests are executed.
- Identify all technical, functional and lead personnel.
Important Performance Testing Terms
Performance Testing is a vast and complex area, where a little mistake can become a blunder and would result in huge loss. Below are the important performance testing terms that will facilitate articulating terms correctly in context of performance engineering.
Throughput
In simple words throughput is the work done by the server in a unit of time. For example, hits per second, requests per second, transactions per second, pages per minute, etc. Below image displays throughput (data transferred in bytes by the server or work done by the server).
Latency
In general, any kind of delay in providing the result is called Latency. Latency is the wasted time. For example, accessing data on a disk, latency is defined as the time it takes to set the read/write head over the correct sector.
Concurrent Users
Concurrent users are the users which are in the system but performing different tasks. For example, there are 50 users in the system, 10 users are logging in, 20 users are checking the products and other 20 are ordering the products.
Simultaneous Users
Simultaneous users are the users which are in the system performing same tasks at the same time. For example, there are 50 users in the system, all the 50 users are ordering the product at the same time. Users hitting the same point together are called simultaneous users.
State Full
A state full connection is one in which connection between two systems is maintained.
State Less
A state less connection is one in which connection between two systems is not maintained.
Cookies
A cookie is a piece of text stored on a user’s computer by web browser. A cookie can be used for authentication, session identification, storing preferences, which can be used to further enhance browsing experience. A cookie consists of one or more name-value pairs containing information, which may be encrypted for information privacy and data security purposes. Cookie is sent or attached as a field in the header of the HTTP response by the server to browser and then it is sent back unchanged by the browser each time server is accessed.
Persistent Cookie
Persistent Cookies are stored in Hard disk. Age of the cookie is defined. If a Persistent Cookie has Max-age set to 1 week, then within a week, the initial value stored in the cookie is sent to the server everytime the request is sent to the server.
Non Persistent Cookie
Non Persistent Cookies are stored in Web Browser. This kind if cookie expires when a users closes the browser or the user remains idle for a a certain period of time.
Response Time
Response time is defined as the elapsed time between the request dispatch (as soon as the request is sent to the server) and receiving response (till the last byte of the response is received by the client). Below screenshot shows the transaction response time for different transactions at different intervals.
Paging
Paging is a kind of memory-management technique by which a computer can store and retrieve data from secondary storage (hard disk) for use in main memory. In the paging memory-management technique, the OS retrieves data from secondary storage in same-size blocks called pages. Paging is an important part of virtual memory implementation in modern operating systems, allowing to use hard disk storage for data that does not fit into RAM.
Connection Pooling
Connection Pooling is a pool of connection which allows to resume the connection without creating a new one. Creating connection object is a time consuming process. If connections are created and destroyed every time, it will have a negative impact on the response time, clients will have to wait longer for the response. Connection poolingis a concept of object reusability. This service is provided by application servers. Below screenshot shows the number of active connections on the server.
Process
Process is defined as the virtual address space in RAM for the execution of a program while threads are a way of splitting a program into two or more simultaneously running tasks. In general, a thread is contained inside a process and different threads in the same process may or mat not share resources.
These were some of the important performance testing terms every performance tester/engineer be aware of. Read the below recommended post to further enhance your knowledge.
RADVIEW
The most flexible and cost effective performance testing software solution for enterprise load, performance, and stress testing.
- Test Creation
- Correlation
- Test Execution
- Analytics
- WebLOAD Dashboard
- Server-side performance
- JavaScript
- Mobile Load Testing
- Technologies Supported
LOADNINJA
The only load testing tool to record, replay, and test in real browsers at scale. Performance testing doesn’t have to be the bottleneck. Cut your load testing time in half by running website performance tests in real browsers at scale. LoadNinja is the only load testing tool to record and replay scripts in minutes, leverage real browsers, and deliver actionable website performance data straight from the browser’s DOM.
Features:
- Reduce Test Script Creation Time by 60%
- Generate Load with 1000s of Real Browsers
- Analyze in Real-Time with VU Inspector and Debugger
- Maximize Test Coverage and Minimize Test Time with Load Test Automation
- Data-Driven Load Tests
- Test Applications in Local Network
SMARTMETER
Performance Testing with Enterprise-Level Features with a familiar jMeter interface.
- Easy-to-use test scenario recorder
- Distributed load testing
- Continuous integration support & test acceptance criteria
- Real time test results on multiple monitors
- Combine load tests with Selenium
- Test environment, monitoring
LOADVIEW-TESTING
Load testing is generating user demand on software or servers to measure response time. Stress test websites, web apps & API with performance testing tools.
- 100% Managed Cloud
- On-Demand Load Tests
- Real Browser Testing
JMETER
The Apache JMeter application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
- Web – HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET, …)
- SOAP / REST Webservices
- FTP
- Database via JDBC
- LDAP
- Message-oriented middleware (MOM) via JMS
- Mail – SMTP(S), POP3(S) and IMAP(S)
- Native commands or shell scripts
- TCP
- Java Objects
MICROFOCUS
Micro Focus offers enterprise application software that provides scalable, industry-leading solutions across Cloud Management, DevOps, Hybrid IT, Security and Risk, and Predictive Analytics.
- Supports a wide range of apps
- Continuous testing
- Root cause analytics
IBM
IBM Rational Performance Tester is a performance testing tool that identifies the presence and cause of system performance bottlenecks. Combining multiple ease-of-use features with granular detail, Rational Performance Tester simplifies the test-creation, load-generation, and data-collection processes that help teams ensure the ability of their applications to accommodate required user loads.
NEOTYS
Automated Load Testing tool enabling you to run Continuous Performance Testing on all your applications & APIs.
- Broad Technology Support
- Scriptless Design and Visual Programming
- Powerful Extractors
- Advanced Correlation
- Check User Paths
- User Path Update
- Prefilled Transaction List for Recording
- Record Here
- Web Secure Ready
- Record Mobile Applications
- Push Mode
- Shared Elements
- Streaming Video and Audio Support
- Advanced Actions
LOADTESTINGTOOL
With a single tool you can test all types of websites, mobile applications, corporate ERP and CRM systems, web API services and even Internet of Things platforms.
- Create tests using a browser or a mobile application
- Remote, distributed and cloud-based testing
- Server and database performance monitoring
- Flexible error handling and adjustable pass/fail criteria
- Charts with instant test results available at run time
- Customizable test report with over 20 tables
- Support of all major web development frameworks
LOADSTER
Loadster is a cloud-hybrid load testing solution for high-performance websites and applications. Load test your sites to find bottlenecks, improve stability, and optimize user experience.
- Load testing from anywhere
- Zero to 60,000 in 2-3 minutes (virtual users, that is)
- Peace of mind pricing
- Get up to speed quickly
- More power for power users
APPVANCE
Appvance.ai is the leader in AI-driven software testing. Appvance IQ, its AI-driven, unified test automation system, transforms testing team results.
- Production Bugs
- Bug-Driven Downtime
- Manual Hours
- Manual Labor Cost