There is no shortage of mobile health (m-health) apps for people facing mental health challenges; one 2018 study estimated 10 000 available for download.Reference Torous, Firth, Huckvale, Larsen, Cosco and Carney1 m-health apps track symptoms, deliver therapy and promote health behaviour interventions. Their success is often evaluated according to user engagement, typically operationalised as frequency and duration of app use, behavioural interaction with the app (for example downloads, clicks) and popularity (for example user reviews, ratings).Reference Fleming, Bavin, Lucassen, Stasiak, Hopkins and Merry2 Usage data is assumed to capture different types and depths of app engagement, yet misses cognitive and emotional responses to the app and is disconnected from behaviour change in real-world settings.Reference Cole-Lewis, Ezeanochie and Turgiss3,Reference Weisel, Fuhrmann, Berking, Baumeister, Cuijpers and Ebert4
User engagement has become an umbrella term used to describe a host of conceptually unique user-centred outcomes, including usability, acceptability, feasibility and satisfaction.Reference Ng, Firth, Minen and Torous5 These different user experiences are all evaluated with the same kinds of usage statistics: dwell time, bounce rates and number of downloads, logins, visits and specific interactions, such as clicking on links, watching videos, completing modules.Reference Fleming, Bavin, Lucassen, Stasiak, Hopkins and Merry2 This creates a schism between the concept of interest and the most salient metrics for its evaluation. It opens the door ‘for [user-engagement indicators] to be selected inappropriately, presented with bias or interpreted incorrectly’ and prevents meaningfully comparisons across apps, studies and user groups.Reference Ng, Firth, Minen and Torous5
m-health engagement is narrowly defined as ‘user uptake … and/or ongoing use, adherence, retention, or completion data’,Reference Fletcher, Foley and Murray6 and thus focuses on quantifying rather than qualifying user engagement. Although it is seemingly objective, easy and unobtrusive to record usage data, studies have documented inconsistencies and called for standardised reporting practices.Reference Fleming, Bavin, Lucassen, Stasiak, Hopkins and Merry2,Reference Ng, Firth, Minen and Torous5,Reference Torous, Lipschitz, Ng and Firth7 In addition, the positive impact of m-health apps on real-world or clinical trial outcomes is inconclusive,Reference Fleming, Bavin, Lucassen, Stasiak, Hopkins and Merry2,Reference Torous, Lipschitz, Ng and Firth7 and the ‘beneficial dose [of apps] … or amount of exposure’ at the population level is unknown.Reference Fleming, Bavin, Lucassen, Stasiak, Hopkins and Merry2 This calls into question which metrics and what thresholds may be indicative of user engagement for different apps, users or mental health conditions.
Steady and sustained app use is typically viewed as positive, disengagement and non-use as negative. This emphasis on user behaviour in user-engagement evaluation misses important information about users and their contexts. User engagement may be influenced by how content is organised and presented, symptom burden, environmental stressors and supports, and the desire for social connectivity.Reference Nicholas, Fogarty, Boydell and Christensen8 These cognitive, emotional and social factors may be evaluated through surveys, interviews and app reviews,Reference Ng, Firth, Minen and Torous5,Reference Nicholas, Fogarty, Boydell and Christensen8 sometimes independent of usage data. Consequently, data sources may be disconnected and unable to inform each other.
It has been proposed that the field of m-health evaluation could be advanced if we understood the relationship between out-of-app and in-app engagement.Reference Cole-Lewis, Ezeanochie and Turgiss3 However, a barrier to this is the way in which in-app user engagement is currently conceptualised and measured.
(Re)Defining user engagement in the m-health space
Human–computer interaction (HCI) researchers define user engagement as capturing and maintaining the attention and interest of technology users,Reference Jacques9 and the cognitive, emotional and temporal investment made by users.Reference O'Brien and Toms10 Mental health interventions target not only behaviour, but emotional and cognitive processes,Reference Perski, Blandford, West and Michie11 and social connectivity may be fundamental for self-management.Reference Nicholas, Fogarty, Boydell and Christensen8 A holistic definition of user engagement with m-health apps goes beyond what people do to how these tools address needs for information/education, social support and personal agency.
HCI researchers have identified attributes of user engagement, including user attention, interest, motivation, control and system usability and aesthetic appeal.Reference Jacques9,Reference O'Brien and Toms10 Focusing on user-engagement attributes offers a targeted approach to measurement. For example, we might use eye-tracking or heat maps to gauge in-app attention, or brief self-report instruments to capture users’ sense of control at defined points in time during an interaction. HCI researchers have modelled user engagement as having natural ebbs and flows over the course of users’ interactions with digital tools. The process of model of engagement suggests that users move through points of engagement, periods of sustained engagement, disengagement and re-engagement and that some attributes are more salient at particular stages than others.Reference O'Brien and Toms10 Process-based models reveal that engagement is not an ‘all or nothing’ phenomenon. Thus, is it essential to not only measure interaction outcomes (i.e. total session duration, total app downloads) but users’ journeys through an app, i.e., ups and downs in interactivity and varying levels of emotional and cognitive involvement.
Qualifying the quantification of m-health app usage
Interpreting the ‘ups and downs’ of in-app user engagement involves looking at usage data differently, and connecting multiple data sources in significant ways. There are ‘unique cognitive, neurological or motor needs arising from mental illness’.Reference Ng, Firth, Minen and Torous5 People with chronic health conditions have fluctuating needs, yet apps do not take into account the diversity of individual lived experiences or different users, for example, young people, those newly diagnosed.Reference Nicholas, Fogarty, Boydell and Christensen8 High usage is not indicative of positive clinical outcomes but may actually reflect worsening mental health. App usage may also exacerbate negative mood by providing poor quality/too much information, technical challengesReference Nicholas, Fogarty, Boydell and Christensen8 or increasing user's awareness of distressing symptoms.Reference Allan, Mcleod, Bradstreet, Beedie, Moir and Gleeson12
Morton et alReference Morton, Hole, Murray, Buzwell and Michalak13 found that people living with bipolar disorder experienced both negative and positive emotions toward self-monitoring; the practice made them feel that they were managing their condition effectively compared with others, but was also ‘an unpleasant reminder that they were living with bipolar disorder’. Nuanced interpretation is not revealed through usage data alone.
Condition-specific knowledge is critical, and participatory design approaches are essential for gathering insights from people with lived experience and healthcare providers. Participatory design draws upon different methods (focus groups, ethnography) to co-create and co-evaluate prototypes with users throughout the design life cycle.Reference Spinuzzi14 Such methods are needed to identify what goals an app should fulfil, how people want to use it and for what purposes, and how design features can reflect user preferences and goals. Such knowledge can aid in the development of appropriate engagement indicators that will facilitate the interpretation of usage and other data sources.
Apps contain different types of content (for example educational articles, quizzes, symptom trackers, social connection tools). Looking at how frequently/long users interact with individual content pages or features may be less informative than grouping features according to function or the need they are intended to serve, such as education, symptom tracking, social support. It may be productive to distinguish content that people typically access once, for example quiz, versus multiple times, for example sleep or exercise logs, and to consider the sequence of content interactions in terms of scaffolding engagement. Rethinking the analysis of usage data would allow for richer interpretations of why people use apps in naturalistic settings, such as for routine maintenance, affirmation, social support. It could also help tailor content according to recovery or illness stage to reduce cognitive load.Reference Morton, Hole, Murray, Buzwell and Michalak13
Algorithms facilitate the identification of engagement patterns based on interactions with apps over time, often based on duration (short versus long term) and frequency (low versus high). Such analyses reveal that different users have different use trajectories, but they do not explain why the pattern is occurring or its significance to clinical outcomes. Diverse and varied streams of data, including user self-reports, discussion forum transcripts, social network data and data from symptom severity, functioning or quality of life measures can be used to make sense of usage data. Cluster analysis is another option, whereby usage patterns are examined in concert with clinical, sociodemographic variables or other data sources, such as in-app text messages between users and coaches to identify reasons for app use/non-use.Reference Chen, Wu, Tomasino, Lattie and Mohr15
Users weigh the benefits of using m-health apps with factors such as usability, convenience, personal risk, for example privacy, and cognitive or emotional effort,Reference Morton, Hole, Murray, Buzwell and Michalak13 which may not be accounted for in in-app assessments. Empirical studies utilising self-report data may not use validated self-report inventories or use them systematically.Reference Ng, Firth, Minen and Torous5 Thus, incorporating different data sources must balance the burden and risk of self-reporting with the benefits, emphasise replicability in the selection and use of self-report instruments, and transparency in data collection and reporting.
Toward a more comprehensive view of in-app engagement
In-app engagement must seek to support the cognitive, emotional and behavioural changes necessary for desired mental health outcomes, including symptom reduction, recovery and quality of life improvements. For this to occur, use/non-use must be connected to broader out-of-app goals, and the value of negative emotions, behaviours and cognitions relative to overall self-management must be considered. For example, non-use may be indicative of improved mental health and the need for less reliance on digital interventions.
It is tempting to want a magic, uniform formula for measuring user engagement, and this has been the appeal of usage data.
We argue instead that user-engagement indices for m-health apps must be:
Corroborative, where different measures including usage data, symptom severity assessment scales and subjective outcome assessments (for example quality of life) are used to determine what meaningful engagement with the app entails.
Outcome, rather than output, oriented. If m-health apps are meant to improve intervention effectiveness, then how they are used becomes more important than how often they are used.
Process based, where we expect to see ebbs and flows in usage. Rather than labelling app users as low or high engagers based on algorithmic (non)-use, we should adopt participatory design approaches (for example journey mapping) to appreciate how different users interact with different features of the app over time.
Expert driven, meaning that the expertise of people with lived experience and clinicians is included throughout the design process to identify salient needs (for example social support) and goals (for example establishing a routine, symptom management) and how these can be met with the app, as well as to inform aesthetic and content design choices.
User-engagement indices should be developed in parallel with the app itself, and draw upon condition-specific knowledge and multiple data sources. The COPE approach (Corroborative, Outcome oriented, Process based, Expert driven) necessitates collaboration among people with mental health conditions, healthcare providers and user-experience designers to develop m-health apps. Documenting each element of this framework would result in greater transparency about how design decisions were made, what is being measured and why, and how the resulting app fits into the broader mental health landscape.
This research is supported by a Canadian Institute for Health Research (CIHR) Project Grant, ‘Bipolar Bridges: A Digital Health Innovation Targeting Quality of Life in Bipolar Disorder’.
Conceptualised and original draft by H.L.O'B; conceptualised and reviewing: E.E.M. and E.M.; critical revision of the manuscript for important intellectual content: all authors.
Declaration of interest
H.L.O'B., E.M., A.K. and S.J.B.: none. E.E.M. discloses grant funding from a private company in the 36 months prior to this publication.
ICMJE forms are in the supplementary material, available online at https://doi.org/10.1192/bjo.2020.72