Quantitative Methods for Experimental Software Engineering Synthesis

Quantitative Methods for Experimental Software Engineering Synthesis

Abstract. Meta-analysis, or quantitative synthesis, is the branch of statistics aimed at combining the results of individual experiments into a single estimate of treatment effect. Meta-analysis is typically performed in the framework of Systematic Literature Reviews, but in SE it is also used to pool together the results of close replications (i.e.: families of experiments) conducted by the same researchers.

There are two broad types of meta-analysis techniques: individual- and aggregated-data. Individual-data meta-analysis uses the data collected for each experimental unit to calculate the global effect size. Usual statistical techniques (e.g.: ANOVA, regression, etc.) are used in this case. Aggregated-data meta-analysis takes a single value per experiment (e.g.: treatments’ averages) instead. We will focus on this second group of techniques.

Aggregated-data meta-analysis techniques can be further divided into continuous and discrete data. Continuous data are the most frequent case in SE and include: Weighted mean difference (e.g.: Glass, Cohen and Hedges’ effect sizes), response ratio (parametric and non-parametric), correlations, vote counting (statistically corrected) and omnibus tests for p-values. We will review one discrete data technique as well: The Mantel-Haenszel method for combining odds ratios.

The tutorial will have a practical orientation. We will introduce the minimal theoretical concepts needed to apply the techniques to some case studies. We will also learn how to use the R environment to make calculations and display the results using forest plots.


Keywords. Weighted mean difference, response, correlation, vote counting, omnibus tests, odds ratio, R