Exhaustive Testing

## What is Exhaustive Testing?

Exhaustive testing is a testing type that falls under functional testing. It is used to verify that the application works successfully under all possible input and precondition combinations. Hence, it ensures that your application will remain fully functional, no matter its scenario.

However, achieving 100% exhaustive testing is practically impossible due to the large number of possible combinations in modern applications. Therefore, it is best to focus on covering as much ground as feasible to improve your application’s robustness and reliability.

Why is Exhaustive Testing Important?

Exhaustive testing is crucial as it aims to uncover every potential defect in a software application by testing all possible input and precondition combinations. Unlike other testing methods that sample only a subset of potential test cases, exhaustive testing ensures the discovery and resolution of even the most obscure bugs.

At a glance, comprehensive testing may seem excessive for small-scaled use and applicability. However, life-critical applications, such as health care, aviation control, and defense systems, can significantly benefit from it by eliminating the risk of bugs and failures.

Steps to Implement Exhaustive Testing

Here are the steps for implementing exhaustive testing for your application. I will explain each step using a healthcare application as an example.

Step 1 – Define the scope and objectives

Begin by clearly outlining what you aim to achieve with exhaustive testing. Identify critical paths, functionalities, and edge cases that need a thorough examination. This ensures that critical functionalities are reliable under all possible scenarios.

For example, a healthcare application should focus on functionalities like patient data entry, medication management, and appointment scheduling.

Step 2 – Gather requirements

Collect all relevant requirements, specifications, and documentation. Understanding the full range of inputs and conditions is vital for creating comprehensive test cases. If we consider data entry functionality, you should have all the information about the input data, such as:

• Valid and invalid data formats – names should be alphabetic, and DOB should be in DD/MM/YYYY format.
• Mandatory fields – patient name, contact number, address.
• Data validation rules – Minimum and maximum length for text fields, valid date ranges.

Step 3 – Design test cases

Develop detailed test cases that cover all possible combinations of inputs and scenarios. You can use combinatorial techniques or tools like Pairwise or Hexawise to ensure every possibility is accounted for.

You need to create test cases for every combination of valid/invalid patient names, dates of birth, medical record numbers, and special cases like missing fields or incorrect data formats for the data entry functionality.

Patient Name:

• Valid: John Doe, Will Smith
• Invalid: 12345, John@Doe

Date of Birth:

• Valid: 01/01/1990, 31/12/2000
• Invalid: 1990/01/01, 01-2000-02

Test Case 1:

• Patient Name: John Doe (Valid)
• Date of Birth: 01/01/1990 (Valid)
• Expected Result: Data entry successful

Test Case 2:

• Patient Name: 12345 (Invalid)
• Date of Birth: 01/01/1990 (Valid)
• Expected Result: Error

Step 4 – Automate where possible

You should always try to automate test execution since a single application can have many input combinations. You can use a testing framework like Selenium or Cypress to automate patient data entry tests.

Step 5 – Execute tests

Run the test cases in an environment that closely mirrors the production environment to ensure accurate results. Ensure all the server configurations, database, and security settings are identical to the production environment.

Step 6 – Analyze results

Thoroughly review the test results to identify defects, such as incorrect handling of invalid data, data loss, or improper validation. This step may involve iterating on the test cases and re-executing them to ensure all issues are addressed.

Types of Exhaustive Testing

Exhaustive testing can be further divided into several types based on the different aspects of the software:

1. Input combination testing: testing all possible combinations of input values.
2. Path testing: validating all possible paths through the application code.
3. State transition testing: testing all possible state transitions within a system.
4. Boundary value testing: testing at the extreme ends of input ranges.
5. Decision table testing: testing different input combinations by creating a decision table.
6. Code coverage testing: focuses on ensuring that every line of code is executed at least once during testing.

Effective Testing vs. Exhaustive Testing

While exhaustive testing provides comprehensive coverage through test types, developers often use another testing strategy called effective testing to address the time and resource constraints in exhaustive testing. The table below will give you a general idea of the differences and similarities between exhaustive and effective testing.

Advantages of Exhaustive Testing

• Comprehensive coverage: Ensures all the possible scenarios are tested.
• High reliability: Increases the reliability of the application by identifying even the most obscure defects.
• Improved quality: Significantly improves the quality of the application by addressing all potential issues.
• Early detection of defects: Identifies defects early in the development cycle, reducing the cost and effort associated with fixing them later.
• Confidence in releases: Developers have more confidence in releases due to strict validations.
• User satisfaction: Improve end-user satisfaction by ensuring a smooth and error-free application.

Disadvantages of Exhaustive Testing

• Resource-intensive: Requires significant time, effort, and computational resources since there are a large number of scenarios to cover.
• Impractical for large systems: Writing test cases for every input combination can be impossible as the applications grow in size and complexity.
• Maintenance overhead: Updating the test suite to changing requirements and software updates can be burdensome.
• Potential redundancy: Some test cases may become redundant, leading to inefficiencies.

Use Cases for Exhaustive Testing

Although 100% exhaustive testing coverage is impossible, it is used in many critical systems to ensure their reliability and robustness.

• Healthcare applications: Healthcare applications handle millions of essential and sensitive patient data records. A minor error in these systems could result in the wrong treatment, medication, or even no diagnosis, endangering the patient. Exhaustive testing ensures that these systems are highly accurate and reliable to improve the quality of healthcare services.
• Aerospace software: Avionics software, including flight control systems, must work without errors and failures in all conceivable situations to protect people’s lives. Even a tiny bug can cause loss of life and significant financial losses. Exhaustive testing ensures software handles all flight scenarios and emergency conditions, guaranteeing safety and reliability.
• Nuclear power plant control systems: Any software failure in these systems can lead to radiation leaks, environmental contamination, and loss of human life. So, it is important to test control software through its extremes so that the software can manage all situations to avoid mishaps and meet safety requirements.

Conclusion

Exhaustive testing covers every possible input and scenario in an application, helping to identify and resolve defects that might otherwise go unnoticed. While achieving 100% exhaustive testing is challenging, the benefits for critical systems and high-risk applications make exhaustive testing essential for ensuring reliability and safety.