> How large are these datasets that have attracted so many research papers?
> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).
> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?
> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.
Is this telling me that most theories about "sw estimation best practices" are cargo cults o-O ?
I think the term 'cargo cult' carries some religious history and other baggage that I wouldn't want conflated here, but your feeling that anything we've heard about software estimation is questionable is probably warranted.
IMO the theories have sprouted more from management which unfortunately is probably the best equipped today to view the trends over time and compare them to actual results (given the lack of empirical data). If we admit that our ideas of estimation have come from management, we should also admit their conflicts of interest in the matter and the various management fads that come and go. I think as humans we all search for patterns and try to generalize rules to solve our problems, even when we have an incomplete view of things to begin with.
Finally, even with a generalized rule we should realize how different people are team-to-team. I haven't worked on two teams where the same exact processes worked for everyone. The best teams and managers I've had will observe a team over time and tweak the various processes according to what works at that point in time. The worst managers were the ones who idolized a specific style or person and copy-pasted their opinions into their workplace without listening to the team.
> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).
> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?
> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.
Is this telling me that most theories about "sw estimation best practices" are cargo cults o-O ?