Skip to main content Accessibility help
×
×
Home

Who watches the watchmen? Evaluating evaluations of El Sistema

  • Geoffrey Baker (a1), Anna Bull (a2) and Mark Taylor (a3)

Abstract

Within the growing field of publications on El Sistema and Sistema-inspired programmes around the world, a marked divide can be observed between the findings of critical academic studies and commissioned evaluations. Using evaluations of El Sistema in Venezuela and Aotearoa New Zealand as our principal case studies, we argue that this gulf can be explained at least partly by methodological problems in the way that some evaluations are carried out. We conclude that many Sistema evaluations display an alignment with advocacy rather than explorative research, and that the foundation for El Sistema's claims of social transformation is thus weak.

Copyright

References

Hide All
ALEMAN, X. et al. (2016) The effects of musical training on child development: A randomized trial of El Sistema in Venezuela. Prevention Science.
BAKER, G. (2014) El Sistema: Orchestrating Venezuela's Youth. New York: Oxford University Press.
BAKER, G. (2016a) Editorial Introduction: El Sistema in critical perspective. Action, Criticism & Theory for Music Education, 15 (1), 1032.
BAKER, G. (2016b) Citizens or subjects? El Sistema in critical perspective. In Elliott, D., Bowman, W., & Silverman, M. (eds.), Artistic Citizenship: Artistry, Social Responsibility, and Ethical Praxis (pp. 313–38). New York: Oxford University Press.
BAKER, G. (2016c) Antes de pasar página: Conectando los mundos paralelos de El Sistema y la investigación crítica. Revista Internacional de Educación Musical, 4, 5160.
BAKER, G. (2017) Big noise in Raploch? Scottish Review, 21 June. http://www.scottishreview.net/GeoffBaker285a.html.
BAKER, G. & FREGA, A. L. (2018) ‘Producing musicians like sausages’: New perspectives on the history and historiography of Venezuela’s El Sistema. Music Education Research. DOI: 10.1080/14613808.2018.1433151
BELFIORE, E. (2002) Art as a means of alleviating social exclusion: Does it really work? A critique of instrumental cultural policies and social impact studies in the UK. International Journal of Cultural Policy, 8 (1), 91106.
BELFIORE, E. (2009) On bullshit in cultural policy practice and research: Notes from the British case. International Journal of Cultural Policy, 15 (3), 343359.
BELFIORE, E. (2016) Cultural policy research in the real world: Curating ‘impact’, facilitating ‘enlightenment’. Cultural Trends, 25 (3), 205216.
BELFIORE, E. & BENNETT, O. (2010) Beyond the ‘toolkit approach’: Arts impact evaluation research and the realities of cultural policy-making. Journal for Cultural Research, 14 (2), 121142.
BORCHERT, G. (2012) Sistema Scotland: A critical inquiry into the implementation of the El Sistema model in Raploch. MMus, University of Glasgow.
BORZACCHINI, C. (2010) Venezuela en el Cielo de los Escenarios. Caracas: Fundación Bancaribe.
BRUNS, S. B. & IOANNIDIS, J. P. A. (2016) p-Curve and p-hacking in observational research. PLoS ONE, 11 (2), e0149144.
BULL, A. (2016) El Sistema as a bourgeois social project: Class, gender, and Victorian values. Action, Criticism, and Theory for Music Education, 15 (1), 120–53.
CID, A. (2014) Giving a second chance: An after-school programme in a shanty town interacted with parent type: Lessons from a randomized trial. Educational Research and Evaluation, 20 (5), 348365.
CUESTA, J. (2011) Music to my ears: The (many) socioeconomic benefits of music training programs. Applied Economics Letter, 18 (10), 915–18.
DE ANGELIS, C. et al. (2004) Clinical trial registration: A statement from the International Committee of Medical Journal Editors. The New England Journal of Medicine, 351, 12501251.
DELATTRE, S. & ROQUAIN, E. (2015) New procedures controlling the false discovery proportion via Romano–Wolf's heuristic. The Annals of Statistics, 43, 11411177.
DOBSON, N. (2016) Hatching plans: Pedagogy and discourse within an El Sistema-inspired program. Action, Criticism, and Theory for Music Education, 15 (1), 89119.
ELPUS, K. (2015) Music teacher licensure candidates in the United States: A demographic profile and analysis of licensure examination scores. Journal of Research in Music Education, 63 (3), 314335.
FINK, R. (2016) Resurrection symphony: El Sistema as ideology in Venezuela and Los Angeles. Action, Criticism, and Theory for Music Education, 15 (1), 3357.
GEN. (2011). Evaluation of Big Noise, Sistema Scotland. Edinburgh: Scottish Government Social Research.
GERRING, J. (2006) Case Study Research: Principles and Practices. Cambridge: Cambridge University Press.
GLASGOW CENTRE FOR POPULATION HEALTH. (2015). Evaluating Sistema Scotland – initial findings report. Glasgow: GCPH.
HEAD, M. L., HOLMAN, L., LANFEAR, R. & JENNIONS, M. D. (2015) The extent and consequences of p-hacking in science. PLoS Biol, 13 (3), e1002106.
HOLLINGER, D. (2006) Instrument of Social Reform: A Case Study of the Venezuelan System of Youth Orchestras. DMA diss., Arizona State University.
HOLOCHWOST, S., WOLF, D. P. & BOSE, J. H. (2017) Building Strengths, Buffering Risk: Evaluating the Effects of El Sistema-Inspired Music Programs in the United States. http://wolfbrown.com/images/books_reports/Building_Strengths_Buffering_Risk.pdf.
JOHANSON, K. & GLOW, H. (2015) A virtuous circle: The positive evaluation phenomenon in arts audience research. Participations, 12 (1), 254270.
LEES, L. & MELHUISH, C. (2015) Arts-led regeneration in the UK: The rhetoric and the evidence on urban social inclusion. European Urban and Regional Studies, 22 (3), 242260.
LOGAN, O. (2015a) Doing well in the eyes of capital: Cultural transformation from Venezuela to Scotland. In McNeish, J.–A., Borchgrevink, A. and Logan, O. (eds.), Contested Powers: The Politics of Energy and Development (pp. 216253). London: Zed Books.
LOGAN, O. (2015b) Hand in Glove: El Sistema and Neoliberal Research. https://www.researchgate.net/publication/287202150.
LOGAN, O. (2016) Lifting the veil: A realist critique of Sistema's upwardly mobile path. Action, Criticism, and Theory for Music Education, 15 (1), 5888.
McKEGG, K., CROCKET, A., GOODWIN, D., & SAUNI, P. (2015) Sistema Aotearoa: Outcomes Evaluation Report. Hamilton: The Knowledge Institute Limited (Kinnect Group).
McPHAIL, G., RATA, E., & SITEINE, A. (2018) The changing nature of music education. In McPhail, G., Thorpe, V. & S. Wise (eds.), Educational Change and the Secondary School Music Curriculum in Aotearoa New Zealand (pp. 7491). London: Routledge.
MERLI, P. (2002) Evaluating the social impact of participation in arts activities: A critical review of François Matarasso's Use or Ornament? International Journal of Cultural Policy, 8 (1), 107118.
NEW ZEALAND LAW COMMISSION. (2001) Māori Custom and Values in New Zealand Law, Study Paper 9, Wellington, N.Z: The Law Commission.
PEDROZA, L. (2015) Of orchestras, mythos, and the idealization of symphonic practice: The Orquesta Sinfónica de Venezuela in the (collateral) history of El Sistema. Latin American Music Review, 36 (1), 6893.
PEREZ, E. & ROJAS, Y. (2013) ¿Por Qué Quiero Que Mi Hijo Sea Músico? Expectativas de las Madres, Cuyos Hijos Están en la OSIC. Diss, Universidad Católica Andrés Bello. http://biblioteca2.ucab.edu.ve/anexos/biblioteca/marc/texto/AAS7348.pdf.
RIMMER, M., STREET, J. & PHILLIPS, T. (2014) Understanding the Cultural Value of In Harmony-Sistema England. Arts & Humanities Research Council/University of East Anglia.
ROSABAL–COTO, G. (2016) Costa Rica's SINEM: A perspective from postcolonial institutional ethnography. Action, Criticism, and Theory for Music Education, 15 (1), 154–87.
SCRUGGS, T. M. (2015) ‘The Sistema,’ the Euroclassical Tradition, and Education as a Transformative Agent to Supercede Class Status. Paper presented at the annual meeting of the Society for Ethnomusicology, Austin, Texas, December 3–6.
SELWOOD, S. (2002) Measuring culture. Spiked Online. http://www.spiked-online.com/newsite/article/6851#.WOd4UGTyugQ
SISTEMA NACIONAL DE ORQUESTAS JUVENILES E INFANTILES. (2011) Evaluación de Impactos. http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=36583351
STUFFLEBEAM, D. L. & CORYN, C. L. S. (2014) Evaluation Theory, Models, and Applications (2nd edition). San Francisco: Jossey–Bass.
THRUPP, M. (2014) At the eye of the storm: Researching schools and their communities enacting National Standards. New Zealand Journal of Educational Studies, 49 (1), 620.
THRUPP, M. & EASTER, A. (2012) Research, Analysis and Insight into National Standards (RAINS) Project: First Report Researching Schools' Enactments of New Zealand's National Standards Policy. http://www.nzei.org.nz/site/nzeite/files/reports/RAINS-Final-2012-03-01.pdf.
THRUPP, M. & WHITE, M. (2013) Research, Analysis and Insight into National Standards (RAINS) Project. Final report: National Standards and the Damage Done. Wellington, New Zealand: The New Zealand Educational Institute Te Riu Roa (NZEI). http://www.education2014.org.nz/.
TRINIK, R. (2014) Window with a view: reflections on Sistema Aotearoa. E-Journal of Studies in Music Edu-cation, 10 (1), 1221. http://www.arts.canterbury.ac.nz/music/merc/downloads/ejournal_v10no1.pdf.
TUHIWAI SMITH, L. (1999) Decolonizing Methodologies: Research and Indigenous Peoples. London: Zed Books.
TUNSTALL, T. (2012) Changing Lives: Gustavo Dudamel, El Sistema, and the Transformative Power of Music. New York: W. W. Norton.
TUNSTALL, T. & BOOTH, E. (2016) Playing for Their Lives: The Global El Sistema Movement for Social Change Through Music. New York: W. W. Norton.
WANG, R. et al. (2007) Statistics in medicine — Reporting of subgroup analyses in clinical trials. The New England Journal of Medicine, 357, 21892194.
WOODSIDE, A. (2010) Case Study Research: Theory, Methods and Practice. Bingley: Emerald Group Publishing.
YIN, R. Y. (2013) Case Study Research: Design and Methods (5th edn). Los Angeles: SAGE Publications.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

British Journal of Music Education
  • ISSN: 0265-0517
  • EISSN: 1469-2104
  • URL: /core/journals/british-journal-of-music-education
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed