Before understanding risk based testing, we must understand the importance of risk based testing by answering following questions.
1)Can we achieve 100% test coverage always?Do we always have enough time to do full regression testing? How much of regression testing has been automated or can be automated?
2)How much % of compatibility testing (browsers, databases, OSs, application servers, networks, hardware platforms) is typically covered during regression testing?
3)How can we assure that the most frequently used modules/components are working fine? Do you use tools to analyze code coverage from manual or automated regression testing?
4)What is the $ value of the time spent in trying to cover 100% regression and compatibility testing vs. risks in releasing the software to customers early due to internal and external factors in the project?
Let’s explore the evolution of risk based testing and what makes it different from other types of testing by answering the above questions. Ideally, no project, and especially no software project, is executed without adverse external influences. We plan to have 100% test coverage on the software products that need to be delivered to the end users. But in a practical world this is seldom possible due to potential problems like delays in requirements and development, non-availability of required test environments on time, budgetary constraints etc. “Why do we need so much time for testing”, “Why do we need to do so many types of testing”, “Reduce the testing time”, “I will take the risk, let us release the software” – are the first reactions by business to IT teams and reduction in testing time is the most common solution adopted by most companies during delays in early phases of software development lifecycles. However, IT teams especially testing teams must learn how to operate and still delivery high quality software under such circumstances.Testing teams can start by prioritizing the tests that need to be performed and this is determined by a set of key people working on the project like the Test Manager, Test Lead with the help of Business Analyst. Though it is a problem based testing technique, it is applied in the case of looming deadlines and limited resources. When combined with the traditional ways to test the application, it limits the scope of the testing efforts. This technique is used to prioritize the development and execution of tests based on the two factors, ‘impact and likelihood of failure’ of the functionality or the aspect being tested. How and Why Risk Based Testing Formulating a risk based testing approach needs an understanding of the risks involved before you start testing the product. There are going to be some residual risks that cannot be eliminated. Risk-Based testing helps to find the right quality (No high severity defects and very less low severity defects) level that can be delivered within limited schedule and resources. This is done by identifying business and technical requirements for an application and prioritizing these requirements on the basis of factors ‘likelihood and impact’ of failure. Risk of a requirement can be assessed based on:
A)Which requirements and attributes are critical for the success of the application/product and the business value of the same based on the time to market. Business Analyst or Project Manager in consultation with the customer can come out with this information.
B)How often is a feature/requirement used?
C)Is there is work around for the user to be able to use the application effectively for this requirement/feature?
D)How visible is a problem in a feature or attribute (for customers, users and people outside)?
E)Can we deliver without this requirement/feature?
Each requirement may result in many test scenarios and hundreds of test cases. Testing team with their deep understanding of business knowledge and end-user perspective must prioritize the test scenarios and test cases, based on risk analysis performed on the requirements and identify the tests that need to be executed. Testing teams should analyze the requirement and test scenarios and score them on following factors based on the business knowledge the team has acquired working on the same project or business area: -Consequence of failure (CF) (For e.g., business impact of failure)
-Probability of failure (PF) (For e.g., size and complexity of a requirement, use of open source components)
-Vulnerability of failure (VF) (for e.g., features/requirements with lot of defects from past test execution cycles)
Scores the requirement on following factors: Functionality,Reliability,Usability,Efficiency,Maintainability, Portability.
Risk based testing is only effective in teams with good business knowledge and end-user perspective.This will determine the risk value of the requirement/test scenario. Even when testing teams do not have sufficient time to write detailed test cases, test scenarios must be documented in detail with key validations to be performed during execution and prioritized. Formalizing this process and incorporating this strategy in your projects will help the testing team deliver the right business deliverables on time and with adequate test coverage. This technique makes it easy for the organizations to deliver the right level of quality in the context of changing business requirements. It provides a critical level of objectivity in determining what things to test by using a combination of business and technical requirements to prioritize the testing process. The challenging part though is how to measure success of risk based testing strategy on a project. There are certain ways in which it can be achieved – Return on Investment (ROI), Delivered defect density, Code coverage analysis and Testing effectiveness.
Why is it Important?
The risk based approach supports the strategy that can expose all the risks and eliminate them. Additionally, it also aims to reduce the impact of residual risks that were not anticipated or eliminated. A complete understanding of the system and its operational environment helps to understand the potential risks thereby establishing a well defined scope of risk identification and assessment thus enabling the testing team to prioritize the test cases to be executed in the order of the high risk to low risk requirements and high risk test cases to low risk test cases for each of the requirements.
To Conclude, Risk Based Testing is an approach that requires skill and experience to isolate the most important tests on the basis of technical and business risks and constraints. Risk based testing is a powerful testing technique that enables the testing teams to streamline their testing efforts. This knowledge helps in mitigating the risk and minimizing the testing efforts, thus, bringing an objectivity to test designing and test management activities. It enables testing teams to make informed decisions while defining exit criteria. It is debatable if the Risk Based Testing is an art or science. It is science because, testing teams follow a technique/structured process to arrive at the test cases to be executed to ensure acceptable quality of the product or application within the constrained time lines. It is an art because deriving the test cases to be executed to obtain acceptable quality depends on the knowledge, skill, experience and judgment of the person prioritizing the tests. For me risk based testing is 50% science and 50% art. What do you think? Please share your thoughts and experiences on risk based testing and whether it is an art or science or combination of both.