Skip to main content Accessibility help

Who watches the watchmen? Evaluating evaluations of El Sistema

Published online by Cambridge University Press:  13 June 2018

Geoffrey Baker
Music Department, Royal Holloway, University of London, Egham, Surrey, TW20
Anna Bull
School of Social, Historical and Literary Studies, University of Portsmouth, Burnaby Road, Portsmouth PO1
Mark Taylor
Sheffield Methods Institute, 219 Portobello, Sheffield, S1 4DP,


Within the growing field of publications on El Sistema and Sistema-inspired programmes around the world, a marked divide can be observed between the findings of critical academic studies and commissioned evaluations. Using evaluations of El Sistema in Venezuela and Aotearoa New Zealand as our principal case studies, we argue that this gulf can be explained at least partly by methodological problems in the way that some evaluations are carried out. We conclude that many Sistema evaluations display an alignment with advocacy rather than explorative research, and that the foundation for El Sistema's claims of social transformation is thus weak.

Research Article
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below.


ALEMAN, X. et al. (2016) The effects of musical training on child development: A randomized trial of El Sistema in Venezuela. Prevention Science.Google Scholar
BAKER, G. (2014) El Sistema: Orchestrating Venezuela's Youth. New York: Oxford University Press.Google Scholar
BAKER, G. (2016a) Editorial Introduction: El Sistema in critical perspective. Action, Criticism & Theory for Music Education, 15 (1), 1032.Google Scholar
BAKER, G. (2016b) Citizens or subjects? El Sistema in critical perspective. In Elliott, D., Bowman, W., & Silverman, M. (eds.), Artistic Citizenship: Artistry, Social Responsibility, and Ethical Praxis (pp. 313–38). New York: Oxford University Press.CrossRefGoogle Scholar
BAKER, G. (2016c) Antes de pasar página: Conectando los mundos paralelos de El Sistema y la investigación crítica. Revista Internacional de Educación Musical, 4, 5160.CrossRefGoogle Scholar
BAKER, G. (2017) Big noise in Raploch? Scottish Review, 21 June. Scholar
BAKER, G. & FREGA, A. L. (2018) ‘Producing musicians like sausages’: New perspectives on the history and historiography of Venezuela’s El Sistema. Music Education Research. DOI: 10.1080/14613808.2018.1433151CrossRefGoogle Scholar
BELFIORE, E. (2002) Art as a means of alleviating social exclusion: Does it really work? A critique of instrumental cultural policies and social impact studies in the UK. International Journal of Cultural Policy, 8 (1), 91106.CrossRefGoogle Scholar
BELFIORE, E. (2009) On bullshit in cultural policy practice and research: Notes from the British case. International Journal of Cultural Policy, 15 (3), 343359.CrossRefGoogle Scholar
BELFIORE, E. (2016) Cultural policy research in the real world: Curating ‘impact’, facilitating ‘enlightenment’. Cultural Trends, 25 (3), 205216.CrossRefGoogle Scholar
BELFIORE, E. & BENNETT, O. (2010) Beyond the ‘toolkit approach’: Arts impact evaluation research and the realities of cultural policy-making. Journal for Cultural Research, 14 (2), 121142.CrossRefGoogle Scholar
BORCHERT, G. (2012) Sistema Scotland: A critical inquiry into the implementation of the El Sistema model in Raploch. MMus, University of Glasgow.Google Scholar
BORZACCHINI, C. (2010) Venezuela en el Cielo de los Escenarios. Caracas: Fundación Bancaribe.Google Scholar
BRUNS, S. B. & IOANNIDIS, J. P. A. (2016) p-Curve and p-hacking in observational research. PLoS ONE, 11 (2), e0149144.CrossRefGoogle ScholarPubMed
BULL, A. (2016) El Sistema as a bourgeois social project: Class, gender, and Victorian values. Action, Criticism, and Theory for Music Education, 15 (1), 120–53.Google Scholar
CID, A. (2014) Giving a second chance: An after-school programme in a shanty town interacted with parent type: Lessons from a randomized trial. Educational Research and Evaluation, 20 (5), 348365.CrossRefGoogle Scholar
CUESTA, J. (2011) Music to my ears: The (many) socioeconomic benefits of music training programs. Applied Economics Letter, 18 (10), 915–18.CrossRefGoogle Scholar
DE ANGELIS, C. et al. (2004) Clinical trial registration: A statement from the International Committee of Medical Journal Editors. The New England Journal of Medicine, 351, 12501251.CrossRefGoogle ScholarPubMed
DELATTRE, S. & ROQUAIN, E. (2015) New procedures controlling the false discovery proportion via Romano–Wolf's heuristic. The Annals of Statistics, 43, 11411177.CrossRefGoogle Scholar
DOBSON, N. (2016) Hatching plans: Pedagogy and discourse within an El Sistema-inspired program. Action, Criticism, and Theory for Music Education, 15 (1), 89119.Google Scholar
ELPUS, K. (2015) Music teacher licensure candidates in the United States: A demographic profile and analysis of licensure examination scores. Journal of Research in Music Education, 63 (3), 314335.CrossRefGoogle Scholar
FINK, R. (2016) Resurrection symphony: El Sistema as ideology in Venezuela and Los Angeles. Action, Criticism, and Theory for Music Education, 15 (1), 3357.Google Scholar
GEN. (2011). Evaluation of Big Noise, Sistema Scotland. Edinburgh: Scottish Government Social Research.Google Scholar
GERRING, J. (2006) Case Study Research: Principles and Practices. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
GLASGOW CENTRE FOR POPULATION HEALTH. (2015). Evaluating Sistema Scotland – initial findings report. Glasgow: GCPH.Google Scholar
HEAD, M. L., HOLMAN, L., LANFEAR, R. & JENNIONS, M. D. (2015) The extent and consequences of p-hacking in science. PLoS Biol, 13 (3), e1002106.CrossRefGoogle Scholar
HOLLINGER, D. (2006) Instrument of Social Reform: A Case Study of the Venezuelan System of Youth Orchestras. DMA diss., Arizona State University.Google Scholar
HOLOCHWOST, S., WOLF, D. P. & BOSE, J. H. (2017) Building Strengths, Buffering Risk: Evaluating the Effects of El Sistema-Inspired Music Programs in the United States. Scholar
JOHANSON, K. & GLOW, H. (2015) A virtuous circle: The positive evaluation phenomenon in arts audience research. Participations, 12 (1), 254270.Google Scholar
LEES, L. & MELHUISH, C. (2015) Arts-led regeneration in the UK: The rhetoric and the evidence on urban social inclusion. European Urban and Regional Studies, 22 (3), 242260.CrossRefGoogle Scholar
LOGAN, O. (2015a) Doing well in the eyes of capital: Cultural transformation from Venezuela to Scotland. In McNeish, J.–A., Borchgrevink, A. and Logan, O. (eds.), Contested Powers: The Politics of Energy and Development (pp. 216253). London: Zed Books.Google Scholar
LOGAN, O. (2015b) Hand in Glove: El Sistema and Neoliberal Research. Scholar
LOGAN, O. (2016) Lifting the veil: A realist critique of Sistema's upwardly mobile path. Action, Criticism, and Theory for Music Education, 15 (1), 5888.Google Scholar
McKEGG, K., CROCKET, A., GOODWIN, D., & SAUNI, P. (2015) Sistema Aotearoa: Outcomes Evaluation Report. Hamilton: The Knowledge Institute Limited (Kinnect Group).Google Scholar
McPHAIL, G., RATA, E., & SITEINE, A. (2018) The changing nature of music education. In McPhail, G., Thorpe, V. & S. Wise (eds.), Educational Change and the Secondary School Music Curriculum in Aotearoa New Zealand (pp. 7491). London: Routledge.Google Scholar
MERLI, P. (2002) Evaluating the social impact of participation in arts activities: A critical review of François Matarasso's Use or Ornament? International Journal of Cultural Policy, 8 (1), 107118.CrossRefGoogle Scholar
NEW ZEALAND LAW COMMISSION. (2001) Māori Custom and Values in New Zealand Law, Study Paper 9, Wellington, N.Z: The Law Commission.Google Scholar
PEDROZA, L. (2015) Of orchestras, mythos, and the idealization of symphonic practice: The Orquesta Sinfónica de Venezuela in the (collateral) history of El Sistema. Latin American Music Review, 36 (1), 6893.CrossRefGoogle Scholar
PEREZ, E. & ROJAS, Y. (2013) ¿Por Qué Quiero Que Mi Hijo Sea Músico? Expectativas de las Madres, Cuyos Hijos Están en la OSIC. Diss, Universidad Católica Andrés Bello. Scholar
RIMMER, M., STREET, J. & PHILLIPS, T. (2014) Understanding the Cultural Value of In Harmony-Sistema England. Arts & Humanities Research Council/University of East Anglia.Google Scholar
ROSABAL–COTO, G. (2016) Costa Rica's SINEM: A perspective from postcolonial institutional ethnography. Action, Criticism, and Theory for Music Education, 15 (1), 154–87.Google Scholar
SCRUGGS, T. M. (2015) ‘The Sistema,’ the Euroclassical Tradition, and Education as a Transformative Agent to Supercede Class Status. Paper presented at the annual meeting of the Society for Ethnomusicology, Austin, Texas, December 3–6.Google Scholar
SELWOOD, S. (2002) Measuring culture. Spiked Online. Scholar
STUFFLEBEAM, D. L. & CORYN, C. L. S. (2014) Evaluation Theory, Models, and Applications (2nd edition). San Francisco: Jossey–Bass.Google Scholar
THRUPP, M. (2014) At the eye of the storm: Researching schools and their communities enacting National Standards. New Zealand Journal of Educational Studies, 49 (1), 620.Google Scholar
THRUPP, M. & EASTER, A. (2012) Research, Analysis and Insight into National Standards (RAINS) Project: First Report Researching Schools' Enactments of New Zealand's National Standards Policy. Scholar
THRUPP, M. & WHITE, M. (2013) Research, Analysis and Insight into National Standards (RAINS) Project. Final report: National Standards and the Damage Done. Wellington, New Zealand: The New Zealand Educational Institute Te Riu Roa (NZEI). Scholar
TRINIK, R. (2014) Window with a view: reflections on Sistema Aotearoa. E-Journal of Studies in Music Edu-cation, 10 (1), 1221. Scholar
TUHIWAI SMITH, L. (1999) Decolonizing Methodologies: Research and Indigenous Peoples. London: Zed Books.Google Scholar
TUNSTALL, T. (2012) Changing Lives: Gustavo Dudamel, El Sistema, and the Transformative Power of Music. New York: W. W. Norton.Google Scholar
TUNSTALL, T. & BOOTH, E. (2016) Playing for Their Lives: The Global El Sistema Movement for Social Change Through Music. New York: W. W. Norton.Google Scholar
WANG, R. et al. (2007) Statistics in medicine — Reporting of subgroup analyses in clinical trials. The New England Journal of Medicine, 357, 21892194.CrossRefGoogle ScholarPubMed
WOODSIDE, A. (2010) Case Study Research: Theory, Methods and Practice. Bingley: Emerald Group Publishing.Google Scholar
YIN, R. Y. (2013) Case Study Research: Design and Methods (5th edn). Los Angeles: SAGE Publications.Google Scholar

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 49
Total number of PDF views: 246 *
View data table for this chart

* Views captured on Cambridge Core between 13th June 2018 - 26th January 2021. This data will be updated every 24 hours.

Hostname: page-component-898fc554b-mscqj Total loading time: 0.48 Render date: 2021-01-26T00:55:23.853Z Query parameters: { "hasAccess": "0", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false }

Send article to Kindle

To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Who watches the watchmen? Evaluating evaluations of El Sistema
Available formats

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Who watches the watchmen? Evaluating evaluations of El Sistema
Available formats

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Who watches the watchmen? Evaluating evaluations of El Sistema
Available formats

Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *