Skip to main content Accessibility help
×
Home

Twitter metrics complement traditional conference evaluations to evaluate knowledge translation at a National Emergency Medicine Conference

  • Stella Yiu (a1), Sebastian Dewhirst (a1), Ali Jalali (a2), A. Curtis Lee (a3) and Jason R Frank (a1) (a4)...

Abstract

Objectives

Conferences are designed for knowledge translation, but traditional conference evaluations are inadequate. We lack studies that explore alternative metrics to traditional evaluation metrics. We sought to determine how traditional evaluation metrics and Twitter metrics performed using data from a conference of the Canadian Association of Emergency Physicians (CAEP).

Methods

This study used a retrospective design to compare social media posts and tradition evaluations related to an annual specialty conference. A post (“tweet”) on the social media platform Twitter was included if it associated with a session. We differentiated original and discussion tweets from retweets. We weighted the numbers of tweets and retweets to comprise a novel Twitter Discussion Index. We extracted the speaker score from the conference evaluation. We performed descriptive statistics and correlation analyses.

Results

Of a total of 3,804 tweets, 2,218 (58.3%) were session-specific. Forty-eight percent (48%) of all sessions received tweets (mean = 11.7 tweets; 95% CI of 0 to 57.5; range, 0–401), with a median Twitter Discussion Index score of 8 (interquartile range, 0 to 27). In the 111 standard presentations, 85 had traditional evaluation metrics and 71 received tweets (p > 0.05), while 57 received both. Twenty (20 of 71; 28%) moderated posters and 44% (40 of 92) posters or oral abstracts received tweets without traditional evaluation metrics. We found no significant correlation between Twitter Discussion Index and traditional evaluation metrics (R = 0.087).

Conclusions

We found no correlation between traditional evaluation metrics and Twitter metrics. However, in many sessions with and without traditional evaluation metrics, audience created real-time tweets to disseminate knowledge. Future conference organizers could use Twitter metrics as a complement to traditional evaluation metrics to evaluate knowledge translation and dissemination.

RésuméObjectif

Les congrès sont conçus pour favoriser l'application des connaissances, mais les méthodes classiques d’évaluation ne conviennent pas vraiment à l'objet visé, et il existe peu d’études sur la recherche d'autres instruments de mesure. La présente étude avait donc comme objectif de comparer la performance des méthodes classiques d’évaluation avec celle d'indicateurs Twitter, à l'aide de données recueillies au cours d'un congrès de l'Association canadienne des médecins d'urgence.

Méthode

Il s'agit d'une étude rétrospective dans laquelle ont été comparées des publications dans les réseaux sociaux et des méthodes classiques d’évaluation en lien avec un congrès annuel de médecine de spécialité. Les publications (« gazouillis ») faites sur la plateforme de réseau social Twitter étaient retenues si elles se rapportaient à une séance. L’équipe a fait la distinction entre les gazouillis originaux et les échanges, et les gazouillis partagés, puis a pondéré le nombre de gazouillis originaux et de gazouillis partagés afin de constituer un nouvel indice d’échanges sur Twitter. Le résultat de l’évaluation des conférenciers a été tiré de l’évaluation du congrès. L’équipe a finalement procédé au calcul de statistiques descriptives et à des analyses de corrélation.

Résultats

Sur un total de 3804 gazouillis, 2218 (58,3%) se rapportaient à des séances en particulier. Quarante-huit pour cent (48%) des séances ont fait l'objet de gazouillis (moyenne : 11,7; IC à 95% : 0- 57,5; plage : 0-401) et le résultat médian de l'indice d’échanges sur Twitter s’élevait à 8 (IIQ : 0-27). Sur les 111 présentations de type usuel, 85 ont été soumises à des évaluations classiques; 71 ont suscité des gazouillis (p > 0,05) et 57 ont fait l'objet et d’évaluations classiques et de gazouillis. Vingt séances d'affichage animées (20 sur 71; 28%) et 44% (40 sur 92) des présentations par affiches ou des présentations orales de résumés ont fait l'objet de gazouillis seuls. Il ne s'est dégagé aucune corrélation significative de la comparaison entre l'indice d’échanges sur Twitter et les instruments classiques d’évaluation (ρ = 0,087).

Conclusion

Aucune corrélation n'a été établie entre les instruments classiques d’évaluation et les indicateurs Twitter. Toutefois, l’équipe a constaté que, durant bon nombre de séances soumises ou non à des mesures classiques d’évaluation, l'assistance envoyait des gazouillis en temps réel pour diffuser des connaissances. Les organisateurs de congrès futurs pourraient donc utiliser des indicateurs Twitter comme instruments complémentaires des méthodes classiques d’évaluation au regard de l'application et de la diffusion des connaissances.

Copyright

Corresponding author

Correspondence to: Dr. Stella Yiu, The Ottawa Hospital, 1053 Carling Avenue, Room M206, Ottawa, OntarioK1Y 4E9Canada; Email: syiu@toh.ca

References

Hide All
1.Davis, D. The case for knowledge translation: shortening the journey from evidence to effect. BMJ 2003;327(7405):33–5.10.1136/bmj.327.7405.33
2.Ioannidis, JP. Are medical conferences useful? And for whom? JAMA 2012;307(12):1257–8.10.1001/jama.2012.360
3.CLocate. Health and medical conferences in Canada 2018. www.clocate.com. 2019. [cited June 14, 2019]. pp. 133. Available at: https://www.clocate.com/ (accessed February 4, 2020).
4.Mansouri, M, Lockyer, J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof 2007;27(1):615.10.1002/chp.88
5.Forsetlund, L, Bjørndal, A, Rashidian, A, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2009;20(2):CD003030.
6.Domino, FJ, Chopra, S, Seligman, M, Sullivan, K, Quirk, ME. The impact on medical practice of commitments to change following CME lectures: a randomized controlled trial. Med Teach 2011;33(9):e495e500.10.3109/0142159X.2011.599452
7.Haustein, S, Peters, I, Sugimoto, CR, Thelwall, M, Larivière, V. Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. J Assoc Inf Sci Techol 2013;65(4):656–69.10.1002/asi.23101
8.Chapman, DD, Aalsburg Wiessner, C, Storberg-Walker, J, Hatcher, T. New learning: the next generation of evaluation? Academia. 2006. Available at: https://www.academia.edu/5119525/New_Learning_The_Next_Generation_of_Evaluation (accessed February 18, 2020).
9.Neves, J, Lavis, JN, Ranson, MK. A scoping review about conference objectives and evaluative practices: how do we get more out of them? Health Res Policy Syst 2012;10(26):26.10.1186/1478-4505-10-26
10.Maloney, S, Tunnecliff, J, Morgan, P, et al. Translating evidence into practice via social media: a mixed-methods study. J Med Internet Res 2015;17(10):e242.10.2196/jmir.4763
11.Mohammadi, E, Thelwall, M, Kwasny, M, Holmes, KL. Academic information on Twitter: a user survey. PLoS One 2018;13(5):e0197265.10.1371/journal.pone.0197265
12.Rosell-Aguilar, F. Twitter: a professional development and community of practice tool for teachers. J Interact Media Educ 2018;(1):6.10.5334/jime.452
13.Coleman, JM, Rice, ML, Wright, VH. Educator communities of practice on Twitter. J Interact Online Learn 2018;16(1):1179.
14.Choo, EK, Ranney, ML, Chan, TM, et al. Twitter as a tool for communication and knowledge exchange in academic medicine: a guide for skeptics and novices. Med Teach 2015;37(5):411–6.10.3109/0142159X.2014.993371
15.Amath, A, Ambacher, K, Leddy, JJ, Wood, TJ, Ramnanan, C. Comparing alternative and traditional dissemination metrics in medical education. Med Educ 2017:51(9):935–41.10.1111/medu.13359
16.Eysenbach, G. Can Tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res 2011;13(4):e123.10.2196/jmir.2012
17.Mishori, R, Levy, B, Donvan, B. Twitter use at a family medicine conference: analyzing #STFM13. Fam Med 2014;46(8):608–14.
18.Anderson, G, Gleeson, S, Rissel, C, Wen, L, Bedford, K. Twitter tweets and twaddle: twittering at AHPA's national health promotion conference. Health Promot J Austr 2014;25(3):143–6.10.1071/HE13112
19.Nomura, JT, Genes, N, Bollinger, HR, Bollinger, M, Reed, JF III. Twitter use during emergency medicine conferences. Am J Emerg Med 2012;30(5):819–20.10.1016/j.ajem.2012.01.020
20.Ross, C, Terras, M, Warwick, C, Welsh, A. Enabled backchannel: conference Twitter use by digital humanists. J Doc 2011;67(2):214–37.10.1108/00220411111109449
21.McKendrick, DR, Cumming, GP, Lee, AJ. Increased use of Twitter at a medical conference: a report and a review of the educational opportunities. J Med Internet Res 2012;14(6):e176.10.2196/jmir.2144
22.Chaudhry, A, Glode, LM, Gillman, M, Miller, RS. Trends in Twitter use by physicians at the American Society of Clinical Oncology Annual Meeting, 2010 and 2011. J Oncol Pract 2012;8(3):173–8.10.1200/JOP.2011.000483
23.Roland, D, May, N, Body, R, Carley, S. Are you a SCEPTIC? SoCial mEdia Precision & uTility In Conferences. Emerg Med J. 2015;32(5):412–3.10.1136/emermed-2014-204216
24.Jalali, A, Wood, T. Analyzing online impact of Canadian Conference of Medical Education through Tweets. EIMJ 2013;5(3). doi: 10.5959/eimj.v5i3.162.
25.Neill, A, Cronin, JJ, Brannigan, D, O'Sullivan, R, Cadogan, M. The impact of social media on a major international emergency medicine conference. Emerg Med J 2014;31(5):401–4.10.1136/emermed-2012-202039
26.Scott, KR, Hsu, CH, Johnson, NJ, Mamtani, M, Conlon, LW, DeRoos, FJ. Integration of social media in emergency medicine residency curriculum. Ann Emerg Med 2014;64(4):396404.10.1016/j.annemergmed.2014.05.030
27.Gricks, B, Woo, HH, Lai, CSL. Tweeting the meeting: an analysis of Twitter activity at the Royal Australasian College of Surgeons Annual Scientific Congress from 2015 to 2018. ANZ J Surg 2019;89(11):1485–9.10.1111/ans.15474
28.Kim, AE, Hansen, HM, Murphy, J, Richards, AK, Duke, J, Allen, JA. Methodological considerations in analyzing Twitter data. J Natl Cancer Inst Monogr 2013;2013(47):140–6.10.1093/jncimonographs/lgt026
29.Maloney, S, Moss, A, Ilic, D. Social media in health professional education: a student perspective on user levels and prospective applications. Adv Health Sci Educ Theory Pract 2014;19(5):687–97.10.1007/s10459-014-9495-7
30.Reinhardt, W, Ebner, M, Beham, G, Costa, C. How people are using Twitter during conferences. In: Hornung-Prahauser, V, Luckmann, M, eds. Creativity and Innovation Competencies on the Web. Salzburg: Salzburg, Austria; 2009:145–6.
31.Scanfeld, D, Scanfeld, V, Larson, EL. Dissemination of health information through social networks: Twitter and antibiotics. Am J Infect Control 2010;38(3):182–8.10.1016/j.ajic.2009.11.004
32.Jalali, A, Sherbino, J, Frank, J, Sutherland, S. Social media and medical education: exploring the potential of Twitter as a learning tool. Int Rev Psychiatry 2015;27(2):140–6.10.3109/09540261.2015.1015502
33.Desai, B. A novel use of Twitter to provide feedback and evaluations. Clin Teach 2014;11(2):141–5.10.1111/tct.12086
34.Bruns, A, Stieglitz, S. Quantitative approaches to comparing communication patterns on Twitter. J Technol Hum Serv 2012;30(3-4):160–85.10.1080/15228835.2012.744249
35.Boyd, D, Golder, S, Lotan, G. Tweet, tweet, retweet: conversational aspects of retweeting on twitter. In: 43rd Hawaii International Conference on System Sciences, Honolulu, HI, 2010.
36.Bakshy, E, Hofman, JM, Mason, WA, Watts, DJ. Everyone's an influencer: quantifying influence on twitter. 2011. Available at: http://snap.stanford.edu/class/cs224w-readings/bakshy11influencers.pdf (accessed February 4, 2020).
37.Brown, A, Riddell, J, Jauregui, J, Yang, J, Nauman, R, Robins, L. What's all the chatter? A mixed-methods analysis of emergency physician's tweets. West J Emerg Med 2017;18(Suppl):S323.
38.Riddell, J, Brown, A, Robins, L, Nauman, R, Yang, J, Jauregui, J. What's all the chatter? A mixed-methods analysis of emergency physician's tweets. West J Emerg Med 2020;21(1).
39.Butler, AC, Raley, ND. The Future of Medical Education: Assessing the Impact of Interventions on Long-Term Retention and Clinical Care. J Grad Med Educ 2015;7(3):483–5.10.4300/JGME-D-15-00236.1
40.Carpenter, CR, Cone, DC. Using publication metrics to highlight academic productivity and research impact. Acad Emerg Med 2014;21:1160–72.10.1111/acem.12482
41.Trueger, NS, Thoma, B, Hsu, CH, Sullivan, D, Peters, L, Lin, M. The Altmetric Score: a new measure for article-level dissemination and impact. Ann Emerg Med 2015;66(5):549–53.10.1016/j.annemergmed.2015.04.022
42.Neiger, BL, Thackeray, R, Van Wagenen, SA, et al. Use of social media in health promotion: purposes, key performance indicators, and evaluation metrics. Health Promot Pract 2012;13(2):159–64.10.1177/1524839911433467
43.Neves, J, Lavis, JN, Ranson, MK. A scoping review about conference objectives and evaluative practices: how do we get more out of them? Health Res Policy Syst 2012;10(1):274.10.1186/1478-4505-10-26

Keywords

Twitter metrics complement traditional conference evaluations to evaluate knowledge translation at a National Emergency Medicine Conference

  • Stella Yiu (a1), Sebastian Dewhirst (a1), Ali Jalali (a2), A. Curtis Lee (a3) and Jason R Frank (a1) (a4)...

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.