Test run: Team decision in software testing

If the decision making process is performed by a group of people rather than a single person or a pure quantitative method when you need to identify one of the best options from a range of choices, consider the common problems that arise. There are many examples of such activities, such as a state resident voting for a governor from multiple candidates, and the board of directors of a large company choosing one of several small companies as an acquisition target. In essence, the team must sort the options in some way to determine the best option.

In the software development and testing process, there are also group decision questions about the best alternative approach. For example, a group of beta users selects the best user interface from many prototypes. Alternatively, an open source project member votes to elect a strategy.

In the software testing community, I found that people were less aware of the actual methods used in implementing the best alternative team decisions and the possible flaws in the process. There are as many as dozens of groups of analytical methods. In this month's column, I'll describe five of the most common methods used in the software test environment for the best alternatives Group decision: Pure relative majority, absolute majority selection, Borda counting System, Condorcet method, and Schulze method.

All of these analytical methods are well known in other areas other than software engineering, and the research on them is very common. When I introduce each method, I will focus on its application in software testing. I am sure you will find that the knowledge of the team's analytical methods is a useful complement to your toolkit.

In many problem domains, there are situations in which a group of people must choose one of several alternative methods. Different problem domains may use different terminology. For example, in sociology, studies of such issues are often referred to as social selection theories, often using terms such as options and evaluators. In political science, the analysis of such issues is often called the electoral theory, which generally uses terms such as candidates and voters. In the field of mathematics, the group decision of the best alternative method is often regarded as a branch of decision theory. Because there is no specific term set for using these methods in the software development and testing environment, I will mix these terms in the introduction.

The methods I have described in this column apply only to a fairly narrow range of issues and are not fully applicable to quantitative situations where the best option has been clearly defined. As you'll see, when a group of people chooses a "best" alternative, the best meaning is defined by the method used. Also, the approach I am introducing here does not apply to a group of collaborative scenarios in which a team of people arrive at a consistent decision through discussion and negotiation.

Pure relative majority and absolute majority run-off

Before we discuss the group decision on the best alternative approach, let's take a look at the simplest and most common approach-but you have to be clear that it's not necessarily the best approach in many test scenarios. Suppose you are developing an enterprise software application that is used internally within your company, and you have created four completely different user interface prototypes. Although one person can be asked to evaluate different prototypes, in many cases a better approach is to require a group of people to prioritize the four different designs.

It turns out that the process of performing team evaluations and interpreting the final results is a bit more cumbersome than you would expect. The simplest possible team evaluation method is to submit the options to the evaluator and allow them to elect or vote on the only alternative method. The option to get the most votes is a winner. This method is often referred to as a pure relative majority rule.