Computer Aided Evaluation (CAA) Development Status and Trends)

Computer Aided Evaluation (CAA) development status and trend

 

I. features and advantages of Computer-Aided Evaluation

Computer Aided Evaluation (CAA) refers to the introduction of computer as a tool or means in the process of evaluating the learner's knowledge, skills and abilities. The terms are roughly the same as those of CAA: copmuter-aided ed testing, computerized assessment, and computer-based assessment) and computer-based testing, both of which belong to the CAA field.

With the development of large-scale social examinations and online education teaching evaluation activities, the demand for CAA is growing, and traditional educational evaluation methods and testing methods are increasingly unable to meet the needs of educational development. Especially when the number of subjects is too large or the geographical location is relatively scattered, and the time, personnel, and funds are insufficient, the traditional testing method is difficult to adapt.

CAA is accepted by the public as a new evaluation method, and has its own characteristics and advantages:

Table 1: features and advantages of CAA

Angle	Features and advantages
Subject	L enable trainees to learn their learning progress through formative evaluation; L summative evaluations enable trainees to perform phased learning; L check whether the current teaching method is valid.
Teaching	L CAA makes the evaluation in a wide range of regions faster and saves time; L save time and make other more targeted evaluation activities possible; L through formative evaluation, you can understand the learning progress of the subjects; L adjust teaching methods by discovering the overall weaknesses of students during the adaptive testing process.
Management	L reduced the time spent on supervision and management, invigilation, and scoring; L reduces errors caused by human factors during the scoring process; L financial and human resources are saved when students from different locations are evaluated at the same time; L paperless examination to reduce the cost of printing; L The test results can be successfully classified and automatically recorded into the trial file; L The database selects questions randomly to effectively avoid cheating.

 

Therefore, it is of great significance to study the application of CAA in education evaluation: it is particularly useful to solve the interaction and feedback problems in the status of separation of teachers and students in Modern Distance Education; it is helpful for teachers to monitor students' learning processes and perform adaptive tests matching students' abilities. For some disciplines, it can automate evaluation and evaluation, this is of great significance for providing the quality of evaluation, ensuring the consistency and impartiality of evaluation, and reducing the evaluation cost. The relevant data obtained by CAA can automatically generate a diagnosis and analysis report, this also has far-reaching significance for the quantitative research on promoting educational evaluation.

Ii. guiding theory of Computer-Aided Evaluation

Currently, CAA mainly uses the CTT classical test theory and irtitem reponse theory to construct a question bank. The core of the two theories is the mathematical model, which is based on different assumptions: the classical measurement theory uses a linear qualitative model; the Project response theory uses a non-linear probability model.

CTT was proposed in the early 20th century. For CTT, the average score and standard deviation of commonly used measurement indicators are shown. CTT has some limitations, which are highlighted in the following aspects:

The CTT is used to evaluate the quality of the test questions (such as difficulty and discrimination), which is heavily dependent on the tested samples. For the same problem, if the tested samples have different levels of competence, the score is very high, on the contrary, the partition value is very low;

● CTT is only applicable to the relative competence level of the tested subjects (the norm reference test) and is not suitable for the actual competence level (the target reference test). For example, in the target reference test, if all the tested messengers cannot pass the test, the difficulty and discrimination cannot be calculated;

● The same set of test questions must be used when the CTT tests a certain ability or knowledge. Otherwise, the test results cannot be directly compared.

IRT is a new testing theory based on the limitations of CTT. IRT includes three aspects: Project reflection model, project reflection model parameter estimation, project and test information function. IRT believes that the responses to the tests are subject to certain psychological traits. The results of the tests and the relationships between these traits can be described by specific project response model curves. At present, it is widely used in the single-dimensional Logistic model proposed by berborm in 1957. pjq can be used

= Qabc Description: The quality parameters of Project A, Project B, and Project C are differentiation parameters, difficulty parameters, and prediction parameters, respectively, indicating the correct response probability of question J. The quality parameter values of a project cannot be obtained through direct measurement. It must be obtained through estimation of the response data. The maximum likelihood estimation is usually used. The introduction of information functions provides information for determining the test accuracy, so that the measurement error can be better controlled based on the capability characteristics of each level.

Although the project response theory has overcome some shortcomings of the classical test theory, there are still many problems that have not been solved. For example, testing relies on a large number of pre-prepared and high-quality questions, however, it is difficult to achieve this in the field of ordinary education. However, there is no doubt that the Project response theory represents the future development direction of the testing theory and practice discussion.

Iii. Development Status of Computer-Aided Evaluation 1. Computer-Aided Testing of paper-based

In the early stages of CAA development, the optical character reader OMR and optical character reader OCR technology were used. OMR can input correct answers to various types of objective questions in the format specified on the machine, and then the reader can modify the volume to display a very powerful processing capability for objective choice questions. In the early stage, OCR adopts the photoelectric reflection method. Now OCR captures the entire image and analyzes and recognizes characters.

2. Single-Host Computer Aided Evaluation PC-CAA

PC-CAA in the evaluation process, the PC is in an isolated state, information and data transmission through the machine to achieve, there is no introduction to the concept of network transmission. This type of CAA system generally has specialized system installation software. PC-CAA is also suitable for one-time large-scale social examination and summative testing. However, it is not suitable for evaluations that require frequent updates to a large number of subjects and question banks.

3. online computer-assisted evaluation WEB-CAA

The WEB-CAA extracts information from the network server database over the Internet or LAN, and then the user tries to make a selection response on the client, and then submits the trial information over the network. Throughout the process, the client is only a tool. After the evaluation, the client does not have any information left, and all the tested information is submitted to the database on the server, then the evaluation system extracts the tested information from the database, obtains the evaluation result, and returns it to the client through web or LAN.

4. Computer Adaptive Test cat

Cat is a test developed on the basis of the Project response theory. The tester believes that the most ideal project to measure a person's ability is a moderately difficult project, that is, the probability of correct or wrong answers is about 0.5. At the beginning of the test, the computer generally gives a difficult question. If the question is correct, the computer will estimate that his ability is higher than the moderate level, and then give him a more difficult question; if he does something wrong, the computer will estimate that his ability is lower than the moderate level, and then give him a question with lower difficulty. Based on the answers to the second question, the computer evaluates its ability. Based on the second estimate, the computer selects the question closest to the estimated value of its ability in the question bank, then, the ability is estimated based on the test response. In this way, as the number of questions being tested increases, the computer's estimation accuracy for his ability is getting higher and higher, and its estimation value will converge to one point, this is a more accurate estimation of the tested capability.

5. Computer Aided recording

In addition to using testing methods for evaluation, you can also obtain related information for evaluation by recording and tracking the behavior of the subjects. This is the research content of computer-aided record, e-works and e-portfolio are among the most influential and widely used computer-aided evaluation strategies.

Iv. Development Trend of Computer-Aided Evaluation 1. Exploration of Computer-Aided Testing of advanced metacognitive Ability

To measure the advanced cognitive ability of the subjects, CAA must design high-quality and high-level questions. The preparation and design of questions is an extremely complex and time-consuming task, involving a lot of content, for example, determining question types, question assessment points, question test attributes (difficulty, discrimination, etc.), setting scores, and organizing exams. At present, the main research is reflected in the transformation of objective questions and the compilation of subjective questions.

To fundamentally solve the problem of a single question type is to break through the "choice ″. Currently, a promising question type is an operational application question.

2. Evaluation of computer-aided formative Teaching

Although the dominant Application Form of CAA in the field of education is computer-aided summative evaluation, there are still a lot of attention and research on Formative Evaluation of computer support. For example, the University of Birmingham used the Triads system mentioned above to evaluate students in the Department of Chemistry and achieved good results.

3. Integration of computer-aided records and Computer-Aided Testing

One of the direct results of CAA applications is that the marking can automatically enter the information management system and the student record database. The integration of computer-aided records and Computer-Aided Testing is a brand new form. On the one hand, we use computer-aided recording to establish the archives of the subjects, and use the environment provided by modern information technology to express some innovative thinking and problem solving results during their normal learning, submitted to the server in the form of electronic works, and then evaluated by a dedicated instructor to give the evaluation results. On the other hand, it is a common CAA system. The testing results are matched and coordinated according to a certain proportion with the records in the files of the subjects, and the final evaluation results are obtained, these results are generally objective and accurate. At present, most of the research is still carried out in the record and evaluation aspects, and there are still few integration of the two.

4. Application of Data Mining and Knowledge Discovery in CAA

The Application of CAA will produce a large amount of evaluation-related data, especially in the context of modern distance education. This data is combined with the teacher-student information database and the data in the teaching process information database as data sources for data mining (KM) and Knowledge Discovery (KDD, it is possible to mine the educational models and rules hidden behind the data to provide a reliable basis for educational decision-making. How to Use the theories and methods of data mining and knowledge discovery to discover and extract the rules and trends hidden in the CAA database will be a new research topic in the CAA field in the future.

5. Research on Data Security of the Online Examination System

The network examination system has the characteristics of a large amount of instantaneous data, high data accuracy and high security requirements. Data Security research mainly covers the system architecture, firewall and system encryption technology.

If the number of subjects is large, the functions are integrated on the server, so that the examination system interface, data access, data storage, and data management are all completed by the server program, the server may become a bottleneck for network database access. Therefore, we should adopt a multi-server model to jointly store, access, and manage data.

In the same way, the traditional network firewall is located between the student examination machine and the server. When a large amount of data passes the examination, because the computer with firewall software needs to analyze all the data packets passed and takes a lot of system time, it is easy to cause Network Data congestion and affect the normal conduct of the examination. Therefore, a server cluster should be used to enable each server in the cluster to have a firewall function, so as to jointly analyze and process network data and effectively balance network load.

In the examination system, there are many confidential links, such as preventing leakage of pre-examination questions, tampering of students' answers after the examination, and interception of data during transmission, to solve these problems, you can use the file encryption technology.