Oracle Problem in Test Execution

by Nataliia Vasylyna | January 30, 2012 10:00 am

Checking of the result is a complex task, because of the theoretical difficulties and practical limitations. According to the long standing theoretical outputs result checking for software testing in common is an insoluble trouble.

To put it differently, there is no algorithmic or entirely automated solution to the common test oracle problem.

From the practical point of view, the expected behavior is hard to describe exactly so that the monitored behavior can be compared against. Software can fail in countless variations. The unexpected behavior may occur in really unexpected ways, making result checking complicated or almost unrealizable.

Nevertheless, there are cases where specific kinds of system failures, such as irresponsive behavior or system crash, are easy to detect.

Let’s try to find a rough solution to the oracle problem:

• In some cases heuristics guesses can be used based on product domain knowledge, for instance, what other equal products would do under equal situations. Therefore, equal products can usually be used as the test oracle to check implementation outputs and to detect system failures.
• The information about execution may be used to link specific behavior to specific program units as well. We can also inspect different product internal data and dynamic implementation state to help decipher the oracle problem. For instance, if an external function is supported by some internal elements, which were not invoked when we conduct software testing for this external function, we can be almost certain there is something incorrect with this test run. Product experts or developers can also help software testers to conduct this complicated task when some significant defect[1] is suspected.
• Different kinds of logicality checking at the time of implementation may be helpful to detect the implementation failures as well.

Related Posts:

• Software Development (Doesn’t) Need Independent QA[2]
• Is Your E-commerce Ready for Black Friday & Cyber Monday? Let’s Check![3]
• Quality Assurance for Renewable Energy Software: How to Do It Right?[4]

Source URL: https://blog.qatestlab.com/2012/01/30/the-oracle-problem-in-the-test-execution/

---