What you know about workload estimation and what you don't know

This article was first published in the IEEE software magazine, presented by infoq and IEEE Computer Society.

More and more evidence shows a trend in which the cost and workload of software projects exceed the limits and are flooded. On average, the flood rate is about 30% [1 ]. In addition, the accuracy of the estimates in the 1980 s and recent surveys shows that there is basically no improvement. (Only Standish Group analyses indicate that the estimation accuracy has been significantly improved. However, the estimation accuracy mentioned in their chaos reports is much more accurate, perhaps only because of the change in their own analysis methods. They used to choose too many problematic project analyses, the selected project is more representative. (2) The estimation method has not changed much. Despite a lot of research into the formal estimation model, "expert estimation" remains dominant. [3]

There is a significant lack of improvement in estimation accuracy, but we know more about workload estimation than before. In this article, I try to summarize some of my knowledge. Some of these knowledge may increase the accuracy of estimation, some tell us what kind of practice cannot bring about improvement, and some are about our known and unknown workload estimation. A complete set of empirical evidence used to draw conclusions can be seen elsewhere. [1]

What we know

After studying the workload Estimation Study, I have chosen seven conclusions with sufficient evidence.

There is no "best" workload estimation model or Method

Many studies compare the accuracy of multiple estimation models and methods. Which of the following is the best choice? [4. The results are unstable because of several core relationships, such as the Development workload and project size, which are also different in different contexts [5 ]. In addition, the greatest factors that affect the development workload seem to be different, which indicates that the estimation model and method should be tailored according to the context.

The core relationship is not stable enough. It can also be used to explain why the advanced estimation model of statistical methods has little or even no improvement in estimation accuracy compared to the simpler model. The advanced statistical method estimation model is closer to historical data. Therefore, when the context changes, it is more accurate than a simple model. This conclusion tells us that software companies should try to build their own estimation models, rather than expecting a common estimation model and tool to be accurate in their environments.

The customer focuses on reducing prices, which is the main cause of heavy workload.

The trend of underestimating workload is the most obvious in price-based competition, such as bidding. When price competition is less important, such as internal software development, this trend is not obvious. In fact, you may even see the opposite result. This shows that one of the main reasons for the high workload is that when a customer chooses a software development supplier, they tend to choose a low price. Therefore, it is more likely that project bids with a low workload will be initiated. These observations show that when selecting a supplier, the customer pays less attention to the price and pays more attention to the capability to avoid excessive workload.

The maximum workload and the minimum workload range are too close.

The gap between the maximum and minimum workload, such as the 90% confidence interval, is too close to reflect the uncertainty in the actual situation. Although there is strong evidence that we cannot accurately set the maximum and minimum workload values, the current estimation method still assumes that this can be done. This is particularly evident in the estimation method (three-point estimation) based on the PERT program evaluation and review technique technology, where the average workload is often calculated based on the minimum and maximum workload.

Software practitioners should use historical data and previous estimation errors to set realistic minimum and maximum workload, rather than expert judgment. [6]

Workload estimation is easy to be biased, and once biased, it is difficult to recover

All software development workload estimates, even if the formal estimation model is used, require expert judgment. However, even if the expert's judgment can be very accurate, it is easy to be biased. The most serious deviation may occur in the following situations: the person responsible for estimation, before or during estimation, learn about the budget, customer expectations, available time, or other so-called "estimated anchor" factors. If you are not aware of it, the estimator's estimation results may be very close to these anchors. For example, knowing that the customer expects a low price or a small number of hours may cause an underestimating workload. If the estimation request contains some guiding words, such as "this project is so small, simple, and how much it will cost", it will mislead experts to estimate.

There is a lot of research into how to recover from misleading, or how to correct estimated deviations, but no reliable method is found. This can be inferred: the person responsible for workload estimation should try to avoid misleading or irrelevant information. For example, the misleading and irrelevant information in the requirement document should be removed.

The relevant historical data and check list can improve the estimation accuracy.

Sufficient evidence shows that the use of historical data and estimation check lists can improve the workload estimation accuracy. When the historical data is related to the project and the inspection list is customized according to the company's situation, it is unlikely that some work will be missed and enough emergency measures will be added to cope with certain risks, previous experiences can also be reused. Therefore, more realistic estimates can be generated. Especially when similar items can be used for analogy or reference class [7] estimation, the estimation accuracy can be improved.

Although historical data (such as the amount of work required for unplanned work and project management) and estimation check lists (such as for easy-to-Forget work reminders) play a role, many companies still do not use any of them, so they cannot improve the accuracy of estimation workload.

Combining Multiple independent estimates can improve estimation accuracy

Compared with the workload estimation, the average estimation accuracy from multiple parties may be more accurate. A key prerequisite for doing so is that the estimation is completed independently. That is to say, the professional knowledge, background, and estimation process of the estimator are different. Similar to Delphi's estimation process, such as "Planning poker", software developers will also present their own estimates (their poker cards ), this is particularly useful in the context of software workload estimation.

The team-based and structured estimation process makes the Mechanical Combination of estimation more valuable, because the sharing of knowledge can increase the total amount of knowledge, such as the total amount of work required to complete a project. Negative impact of group estimation judgment, such as "collective thinking" and taking on more risks in the group, which has not yet been seen in relevant documents of software workload estimation.

Generally, the estimation model is less accurate than the expert estimation. However, if combined, the difference between the model and the experts may increase the estimation accuracy.

Estimation has its own harm

Estimation not only predicts the future, but also frequently affects the future. Too low estimates can lead to too low quality and may need to be reworked in the future; too high estimates may reduce productivity and follow the Parkinson's Law-work will automatically take up all the available time.

Therefore, you must consider whether workload estimation is required. If it is dispensable or not necessary in the future, it may not be better, or the estimation may be postponed until more information is available. Agile Software Development-only estimates the workload of the next sprint or release version, and must use feedback from previous sprint or version releases-may be a good way to avoid potential hazards of premature estimates.

What we don't know

There are some problems in estimation activities. We just cannot find a good solution, even if we do more research. Three challenges show that our knowledge is far from satisfying us.

How to accurately estimate the workload of ultra-large scale and complex projects

Ultra-large projects impose higher requirements on workload estimation. Not only are more values at risk, but also less experience or historical data. Typical activities in many super-large projects, such as organizational issues-too many project stakeholders are involved, which is difficult to estimate because process changes are often involved, and complex interactions between project stakeholders and existing software applications.

How to accurately estimate the Software size and complexity

Although the measurement of software scale and complexity has been studied for years, when it comes to estimation of Software Development workload, no method is particularly effective. Some context of software scale and complexity may produce accurate estimates, but this is rare.

How to measure and predict Work Efficiency

Even if you can well estimate the size and complexity of the software, you still need to reliably predict the efficiency of individual or team work. This kind of prediction is complicated because the efficiency gap between different software developers and teams is large. There is no good prediction method, except for some practical programming tests (such as trialsourcing.

Currently, we do not even know whether software projects have the "economies of scale" effect or "economies of scale" effect. Many empirical studies show that general software projects have economies of scale, but software practitioners basically believe that "economies of scale ". However, the results of research on economies of scale seem to depend on the implementation method of the analysis, and the relationship between scale and work efficiency is not revealed.

Our current understanding of software workload and cost estimation basically cannot solve the estimation challenges in the software industry. However, it does point out a variety of measures to improve estimation accuracy. In particular, if software companies are able to perform the following actions, they can increase the efficiency of estimation:

• A simple estimation model should be developed and used based on actual conditions and used together with expert estimation.
• Use the historical estimation error to set the maximum-Minimum workload interval.
• Avoid exposure to misleading and unrelated estimation information.
• Use the adjusted inspection list for the Organization.
• To use a structured, group-based estimation process, we must ensure the independence of the estimation.
• Avoid early estimation based on highly incomplete information.

In highly competitive bidding rounds, it is easy for customers to focus on low prices, which can easily lead to over-optimism of the bidder, resulting in high costs and low software quality. This is known as the "Curse of winners" in other fields ". In the long run, many software customers will begin to understand that their opinions on fixed costs and low prices of software projects will negatively affect project success. Prior to that, software companies should be aware that they were chosen when they were in this situation because they were too optimistic about the cost; at this point they had to make strategic preparations to manage or avoid the curse of the winner.

References

1. T. halkjelsvik and M. J? Rgensen, "From origami to software development: A Review of Studies on judgment-based predictions of performance time," Psychological Bulletin, vol. 138, no. 2, 2012, pp. 238-271.
2. M. J? Rgensen and K. mol? Kken -? Stvold, "How large are software cost overruns? A review of the 1994 chaos Report, "information and software technology, vol. 48, No. 4, 2006, pp. 297-301.
3. M. J? Rgensen, "A Review of Studies on expert estimation of software development effort," J. systems and software, vol. 70, no. 1, 2004, pp. 37-60.
4. T. menzies and M. xie Perd, "Special Issue on repeatable results in software engineering prediction," empirical software Eng ., vol. 17, No. 1, 2012, pp. 1-17.
5. J.j. dolado, "on the problem of the software cost function," information and software technology, vol. 43, no. 1, 2001, pp. 61-72.
6. M. J? Rgensen and D. I. K. SJ? Berg, "An Effort Prediction Interval approach based on the empirical distribution of previous estimation accuracy," information and software technology, vol. 45, no. 3, 2003, pp. 123-136.
7. B. flyvbjerg, "Curbing optimism bias and strategic misrepresentation in planning: Reference class forecasting in practice," European Planning Studies, vol. 16, no. 1, 2008, pp. 3-21.

What you know about workload estimation and what you don't know