Evidence-based treatment decisions depend on the accumulation of empirical data from individual studies. Individua studies use a variety of methods with variable quality, reflect random fluctuations and systematic biases, and may yield inconsistent findings. Meta-analysis offers a statistical approach to pool relevant studies together, which can reduce the effect of random error and bias, and produce more reliable effect estimates than individual studies. Despite the important contributions that this method provided for evidence-based medicine, it has been criticised as “statistical alchemy for the 21st century” and described as “new bete noir” which should be “stiffled at birth”. Some of this controversy has been driven by poor practice: the arbitrary pooling of dissimilar studies with unrelated outcomes, the application of statistical technique without sufficient expertise, and with insufficient attention to the clinical context. However, controversy also arises from methodological limitations of the meta-analytic approach, which need to be critically examined in order to evaluate the findings. In this presentation, we examine the limitations of meta-analyses both in terms of current procedures (e.g., biased selection, lack of prospective planning, increased likelihood of chance findings due to multiple testing) and methodological shortcomings (e.g., limited ability to handle multivariate outcomes or to incorporate covariates in the meta-analytic model simultaneously at the level of the study, treatment arm and the individual). By bringing together the weaknesses in a systematic way, it is hoped to foster a more reliable and critical appraisal of the empirical evidence both by researchers and clinicians, which will improve treatment decisions.