Skip to main content Accessibility help


  • Magdalena Ruth Moshi (a1), Rebecca Tooher (a2) and Tracy Merlin (a3)



To identify and appraise existing evaluation frameworks for mobile medical applications (MMA) and determine their suitability for use in health technology assessment (HTA) of these technologies.


Systematic searches were conducted of seven bibliographic databases to identify literature published between 2008 and 2016 on MMA evaluation frameworks. Frameworks were eligible if they were used to evaluate at least one of the HTA domains of effectiveness, safety, and/or cost and cost-effectiveness of an MMA. After inclusion, the frameworks were reviewed to determine the number and extent to which other elements of an HTA were addressed by the framework.


A total of forty-five frameworks were identified that assessed MMAs. All frameworks assessed whether the app was effective. Of the thirty-four frameworks that examined safety, only seven overtly evaluated potential harms from the MMA (e.g., the impact of inaccurate information). Only one framework explicitly considered a comparator. Technology specific domains were sporadically addressed.


None of the evaluation frameworks could be used, unaltered, to guide the HTA of MMAs. To use these frameworks in HTA they would need to identify relevant comparators, improve assessments of harms and consider the ongoing effect of software updates on the safety and effectiveness of MMAs. Attention should also be paid to ethical issues, such as data privacy, and technology specific characteristics. Implications: Existing MMA evaluation frameworks are not suitable for use in HTA. Further research is needed before an MMA evaluation framework can be developed that will adequately inform policy makers.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Available formats

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Available formats

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Available formats



Hide All

The authors thank David Tamblyn for assisting in the trialing and testing of the standardization tool. Magdalena Moshi is a recipient of an Australian Government Research Training Program Scholarship.



Hide All
1.Powell, AC, Landman, AB, Bates, DW. In search of a few good apps. JAMA. 2014;311:1851-1852.
2.Royal Australian College of General Practice. 1.3 m-Health: Royal Australian College of Genral Practice; [cited April 6, 2016]. (accessed August 6, 2018).
3.Laranjo, L, Lau, A, Oldenburg, B, et al. mHealth technologies for chronic disease prevention and managemnet. An evidence check review brokered by the Sax Institute for Healthdirect Australia Sax Institute. Ultimo, NSW: Sax Institute for Helathdidrect Australila; 2015.
4.Dolan, B. WellDoc's BlueStar secures first mobile health reimbursement: Mobihealthnews; 2013 [cited April 29, 2016]. (accessed August 6, 2018).
5.Paris, V, Devaus, M, Wei, L. Health systems institutional characteristics: A survey of 29 OECD countries. OECD Health Working Papers No 50. OECD Publishing 2010.
6.Dolan, B. Caterna offers prescribable, reimbursed eye-strengthening gaming app in Germany: Mobihealthnews; 2014 [cited April 29, 2016]. (accessed August 6, 2018).
7.Clifford, R. NICE to bring HTA processes to mobile health apps: PM Group Worldwide Limited 2017 [cited May 1, 2017]. (accessed August 6, 2018).
8.Mulryne, J, Clemence, B. NICE to introduce HTA-style app assessment process from March 2017: EU Regulatory and Litigation; 2017 [cited May 1, 2017]. (accessed August 6, 2018).
9.Donker, T, Petrie, K, Proudfoot, J, Clarke, J, Birch, MR, Christensen, H. Smartphones for smarter delivery of mental health programs: A systematic review. J Med Internet Res. 2013;15:e247.
10.Busse, R, Orvain, J, Velasco, M, et al. Best practice in undertaking and reporting health technology assessments - Working group 4 report. Int J Technol Assess Health Care. 2002;18:361-422.
11.Merlin, T, Tamblyn, D, Ellery, B, Group, IQA. What's in a name? Developing definitions for common health technology assessment product types of the International Network of Agencies for Health Technology Assessment (inahta). Int J Technol Assess Health Care. 2014;30:430-437.
12.Robustillo Cortés, Me, Cantudo Cuenca, MR, Morillo Verdugo, R, Calvo Cidoncha, E. High quantity but limited quality in healthcare applications intended for HIV-infected patients. Telemed J E Health. 2014;20:729-735.
13.Stoyanov, SR, Hides, L, Kavanagh, DJ, et al. Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3:e27.
14.Powell, AC, Torous, J, Chan, S, et al. Interrater Reliability of mHealth app rating measures: Analysis of top depression and smoking cessation apps. JMIR Mhealth Uhealth. 2016;4:e15.
15.Beatty, AL, Fukuoka, Y, Whooley, MA. Using mobile technology for cardiac rehabilitation: A review and framework for development and evaluation. J Am Heart Assoc. 2013;2:e000568.
16.Drincic, A, Prahalad, P, Greenwood, D, Klonoff, DC. Evidence-based mobile medical applications in diabetes. Endocrinol Metab Clin North Am. 2016;45:943-965.
17.Portelli, P, Eldred, C. A quality review of smartphone applications for the management of pain. Br J Pain. 2016;10:135-140.
18.Chan, S, Torous, J, Hinton, L, Yellowlees, P. Towards a framework for evaluating mobile mental health apps. Telemed J E Health. 2015;21:1038-1041.
19.McMillan, B, Hickey, E, Patel, MG, Mitchell, C. Quality assessment of a sample of mobile app-based health behavior change interventions using a tool based on the National Institute of Health and Care Excellence behavior change guidance. Patient Educ Couns. 2016;99:429-435.
20.Albrecht, UV, Von Jan, U, Pramann, O. Standard reporting for medical apps. In: Mantas, J, Hasman, A, editors. Informatics, management and technology in healthcare. 190: IOC Press; 2013:201-203.
21.Kassianos, AP, Emery, JD, Murchie, P, Walter, FM. Smartphone applications for melanoma detection by community, patient and generalist clinician users: A review. Br J Dermatol. 2015;172:1507-1518.
22.Pandey, A, Hasan, S, Dubey, D, Sarangi, S. Smartphone apps as a source of cancer information: Changing trends in health information-seeking behavior. J Cancer Educ. 2013;28:138-142.
23.Yasini, M, Marchand, G. Mobile health applications, in the absence of an authentic regulation, does the usability score correlate with a better medical reliability? Stud Health Technol Inform. 2015;216:127-131.
24.Murfin, M. Know your apps: An evidence-based approach to evaluation of mobile clinical applications. J Physician Assist Educ. 2013;24:38-40.
25.Gibbs, J, Gkatzidou, V, Tickle, L, et al. 'Can you recommend any good STI apps?' A review of content, accuracy and comprehensiveness of current mobile medical applications for STIs and related genital infections. Sex Transm Infect. 2017;93:234-235.
26.Grundy, QH, Wang, Z, Bero, LA. Challenges in assessing mobile health app quality: A systematic review of prevalent and innovative methods. Am J Prev Med. 2016;51:1051-1059.
27.Huckvale, K, Morrison, C, Ouyang, J, Ghaghda, A, Car, J. The evolution of mobile apps for asthma: An updated systematic assessment of content and tools. BMC Med. 2015;13:58.
28.Martínez-Pérez, B, De La Torre-Díez, I, Candelas-Plasencia, S, López-Coronado, M. Development and evaluation of tools for measuring the quality of experience (QoE) in mHealth applications. J Med Syst. 2013;37:9976.
29.Martinez-Perez, B, de la Torre-Diez, I, Lopez-Coronado, M. Experiences and results of applying tools for assessing the quality of a mHealth app named Heartkeeper. J Med Syst. 2015;39:142.
30.Gautham, M, Iyengar, MS, Johnson, CW. Mobile phone-based clinical guidance for rural health providers in India. Health Informatics J. 2015;21:253-266.
31.mHIMSS App Usability Work Group. Selecting a mobile app: Evaluating the usability of medical applications. Chicago, IL: Healthcare Infomation and Management Systems Society; 2012.
32.Aungst, TD, Clauson, KA, Misra, S, Lewis, TL, Husain, I. How to identify, assess and utilise mobile medical applications in clinical practice. Int J Clin Pract. 2014;68:155-162.
33.Fairburn, CG, Rothwell, ER. Apps and eating disorders: A systematic clinical appraisal. Int J Eat Disord. 2015;48:1038-1046.
34.Hacking Medicine Institute. RANKED Hacking Medicine Instiute; [cited February 27, 2017]. (accessed August 8, 2018).
35.Merlin, T, Lehman, S, Hiller, JE, Ryan, P. The “linked evidence approach” to assess medical tests: A critical analysis. Int J Technol Assess Health Care. 2013;29:343-350.
36.Ferrero-Alvarez-Rementeria, J, Santana-Lopez, V, Escobar-Ubreva, A, Vazquez-Vazquez, M. Quality and safety strategy for mobile health applications: A certification programme. Eur J ePractice. 2013;21:14-26.
37.Singh, K, Drouin, K, Newmark, LP, et al. Developing a framework for evaluating the patient engagement, quality, and safety of mobile health applications. Issue brief (Commonw Fund). 2016;5:1-11.
38.Anxiety and Depression Association of America. Mental health apps: Anxiety and depression association of America; 2016 [cited March 5, 2017].
39.Brooks, GC, Vittinghoff, E, Iyer, S, et al. Accuracy and usability of a self-administered 6-minute walk test smartphone application. Circ Heart Fail. 2015;8:905-913.
40.Walsworth, DT. Medical apps: Making your mobile device a medical device. Fam Pract Manag. 2012;19:10-13.
41.Basilico, A, Marceglia, S, Bonacina, S, Pinciroli, F. Advising patients on selecting trustful apps for diabetes self-care. Comput Biol Med. 2016;71:86-96.
42.Lee, H, Sullivan, SJ, Schneiders, AG, et al. Smartphone and tablet apps for concussion road warriors (team clinicians): A systematic review for practical users. Br J Sports Med. 2015;49:499-505.
43.Reynoldson, C, Stones, C, Allsop, M, et al. Assessing the quality and usability of smartphone apps for pain self-management. Pain Med. 2014;15:898-909.
44.Schnall, R, Mosley, JP, Iribarren, SJ, et al. Comparison of a user-centered design, self-management app to existing mHealth apps for persons living with HIV. JMIR Mhealth Uhealth. 2015;3:e91.
45.Shaia, KL, Farag, S, Chyjek, K, Knopman, J, Chen, KT. An evaluation of mobile applications for reproductive endocrinology and infertility providers. Telemed J E Health. 2017;23:254-258.
46.Shen, N, Levitan, MJ, Johnson, A, et al. Finding a depression app: A review and content analysis of the depression app marketplace. JMIR Mhealth Uhealth. 2015;3:e16.
47.Lalloo, C, Jibb, LA, Rivera, J, Agarwal, A, Stinson, JN. “There's a pain app for that”: Review of patient-targeted smartphone applications for pain management. Clin J Pain. 2015;31:557-563.
48.International Medical Device Regulators Forum. Draft: Software as a Medical Device (SaMD): Clinical evaluation. International Medical Device Regulators Forum; 2016.
49.Medical Services Advisory Committee (MSAC). Guidelines for the assessment of diagnostic technologies. Canberra, ACT: Commonwealth of Australia 2005.
50.Medical Services Advisory Committee. Technical guidelines for preparingassessment reports for the Medical Services Advisory Committee – Medical Service Type: Therapeutic. Australian Government; 2016.
51.Medical Services Advisory Committee. Technical guidelines for preparing assessment reports for the Medical Services Advisory Committee – Service Type: Investigative. Australian Government; 2016.
52.U.S. Food and Drug Administration. Cybersecurity: U.S. Department of Health and Human Services; 2017 [cited June 6, 2017]. (accessed August 8, 2018).
53.U.S. Food and Drug Administration. FDA fact sheet. The FDA's roll in medical device cybersecurity. Washington, DC: U.S. Department of Health and Human Services; 2017.
54.International Medical Device Regulators Forum. “Software as a medical device”: Possible framework for risk categorization and corresponding considerations. International Medical Device Regulators Forum; 2014.
55.Forbrukerradet. Consumer protection in fitness wearables. Norway: Forbrukerradet; 2016.
56.O'Meley, PL, ed. Update on software as a medical device (SaMD). ARCS Scientific Congress 2016; Canberra: Australian Government.
57.Arnhold, M, Quade, M, Kirch, W. Mobile applications for diabetics: A systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res. 2014;16:e104.
58.BinDhim, NF, Hawkey, A, Trevena, L. A systematic review of quality assessment methods for smartphone health apps. Telemed J E Health. 2015;21:97-104.
59.Chomutare, T, Fernandez-Luque, L, Arsand, E, Hartvigsen, G. Features of mobile diabetes applications: Review of the literature and analysis of current applications compared against evidence-based guidelines. J Med Internet Res. 2011;13:e65.
60.Demidowich, AP, Lu, K, Tamler, R, Bloomgarden, Z. An evaluation of diabetes self-management applications for Android smartphones. J Telemed Telecare. 2012;18:235-238.
61.Hoppe, CD, Cade, JE, Carter, M. An evaluation of diabetes targeted apps for Android smartphone in relation to behaviour change techniques. J Hum Nutr Diet. 2017;30:326-338.
62.IMS Institute for Healthcare Informatics. Patient apps for improved healthcare from novelty to mainstream. IMS Instutute for Healthcare Informatics; 2013.
63.Jin, M, Kim, J. Development and evaluation of an evaluation tool for healthcare smartphone applications. Telemed J E Health. 2015;21:831-837.
64.Mobasheri, MH, Johnston, M, King, D, Leff, D, Thiruchelvam, P, Darzi, A. Smartphone breast applications - What's the evidence? Breast. 2014;23:683-689.
65.PsyberGuide. PsyberGuide Ratings- Explanation: PsyberGuide; [cited March 5, 2017].
66.Shah, N, Jonassaint, J, De Castro, L. Patients welcome the sickle cell disease mobile application to record symptoms via technology (SMART). Hemoglobin. 2014;38:99-103.
67.Social Wealth. Our approach: Social wealth. 2016 [cited February 27, 2017].


Type Description Title
Supplementary materials

Moshi et al. supplementary material
Moshi et al. supplementary material 1

 Word (19 KB)
19 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed