Test policy of the expert system: not verifiable?

--- The expert system brings different challenges to testers.

Most of the software we come into contact with is common software or enterprise applications. It is easy to use and easy to understand. correspondingly, there is a kind of software called an expert system that assists experts in a certain field in making judgments and decisions. for example, geological monitoring, weather forecasting, and various simulation software used by scientists we often see in movies: tornado Disaster simulation, Professor metamorphosis's molecular chemistry and new drug experiments.

Intuitively, this type of software is different from the software we use. Specifically, from the perspective of software testing, we can understand the challenges that expert systems bring to us different from general software:

• The user must have the necessary professional knowledge, and this threshold is usually not low.

• Algorithms may require a large amount of input, and data is not well prepared.

• The input space and result space of an algorithm are too large, and there is no intuitive correspondence between them. That is, the output can be seen at a glance for a given input.

• Some types of expert systems have complex interface display, which is far from comparable to common enterprise applications and personal software.

I have not found much information about this. I just want to summarize my previous experiences and discussions on Weibo.

 

1. the test problem is essentially a design problem. testability is an important design constraint. represents the separation of logic and business logic, which is more necessary and urgent in complex systems. all calculation and inference algorithms should be independent of the UI. the UI is only responsible for processing user input and displaying computing results. in this way, from the test point of view, the core logic can be tested without the UI, so the focus of the UI-based testing can be transferred to the verification of user interaction rather than business-relatedBut the convenience of UI testing in other types of applications reduces the urgency of its needs.

2. the test question is essentially a knowledge question. testers must have the knowledge to judge right or wrong. this seems to be common sense, but in the testing of the expert system, this is not always well satisfied. when software is small, we can teach experts to program themselves to compile the software they need. when the software is large,We can recruit professionals rather than software professionals to help testTry. If none of these conditions are met,We can only train the current testers to gain field knowledge.

3. one constraint of expert system evolution is that its computing results should not deteriorate, either unchanged or more accurate. this makes benchmark-based testing possible. after the software runs stably for a period of time, we can collect input and output that a group of users think are correct. as a benchmark, subsequent version changes should not produce different results, unless the algorithm is changed. this is a form of regression, but its application scenarios are relatively limited. After all, the core of the expert system is the algorithm, and the new feature is mostly the enhancement of the algorithm. in this way, the benchmark test will not work.

4. How can we test algorithms? I think this is the most difficult part of expert system testing. the difficulty lies in that the input space and result space of the algorithm are very extensive, and the association between the input and output of the actual algorithm is not like the square or Fibonacci sequence.Obviously, the good news is that in the current computer architecture, algorithms are always deterministic and normalized, that is, given input, you can always get the same output. this is still a design problem: in small cases, the decomposition of algorithms; in large cases, the architecture,For example, the division of duties between the knowledge base and the inference engine. We can always test the implementation of simple mathematical formulas. If we can break down complex algorithms into combinations of simple algorithms,We can at least ensure the intuition and comprehensibility of each test step.
 

There is an important difference: verifying the correctness of the algorithm and its implementation.

The algorithm itself may be purely mathematical and has nothing to do with programming. It is solution rather than implementation. in this sense, the correctness is beyond the scope of our discussion today. It should be ensured by experts. the Method for Testing algorithms based on historical data mentioned in the microblog discussion is the algorithm itself, rather than the implementation of algorithms. we focus on the correctness of the implementation. several specific policies.

• Benchmark Test. we mentioned earlier.

• Qualitative over quantitative. quantification is relatively difficult, because the characteristics of such software determine the uncertainty of the results. however, the nature of the correct result, such as positive number, negative number, and order of magnitude, can be determined.

• Range. An actual example of qualitative over quantitative. it is acceptable that the result falls within the specified range.

 

Conclusion:

• Testers must have field knowledge. We do not feel so necessary on a daily basis, but this rule is applicable when the field threshold is low.An approximation.

• It indicates that the logic must be separated from the business logic. We are not so strict and have not suffered too much retribution. It is only easier to test the UI.A kind of luck.

• The correctness of the algorithm itself is tested separately from that of the Computer Implementation of the algorithm. the former depends on people who have mastered the relevant knowledge (usually experts). The latter is easier to test (usually QA) and automation.

 

Appendix: Expert System Introduction

According to Wikipedia:

"Expert systems are an important branch of early artificial intelligence. They can be considered as a kind of computer Intelligent Program System with special knowledge and experience, generally, Knowledge Representation and Knowledge Reasoning technologies in artificial intelligence are used to simulate complex problems that can be solved by field experts.

Generally, the expert system = knowledge base + inference engine. Therefore, the expert system is also known as a knowledge-based system. An expert system must have three elements:

1. Expert knowledge in the field

2. Simulate expert thinking

3. Expert Level

Expert systems are suitable for the diagnosis, interpretation, monitoring, prediction, planning and design of unrecognized theories and methods, inaccurate data or incomplete information, a shortage of human experts, or very expensive expertise. Generally, knowledge-intensive tasks are implemented by expert systems ."

Figure of an expert system found on the Internet: