Skip Navigation

General Preventive Medicine Residency

Evidence Warehouse

There are thousands of mobile health applications, but evidence for their efficacy is uncertain. In an environment where patients, physicians and payers recognize the potential of technology to change health, how do we know which applications are safe? How do we know which are effective and efficacious?

Studies have highlighted the lack of utility, efficacy, effectiveness, and safety of a large number of apps. These studies recommend end-user engagement in the development of health technologies and make a case for the involvement of healthcare workers in the early development process. Some studies point out the predicament of studying health technologies where the period of study can exceed the life-time of the app or the particular version begin studied, rendering the findings antiquated at time of publication (see Kumar et al, 2013). For all stakeholders in this space—start-ups, clinicians, consumers, and others—emerging technologies are presenting opportunities for innovation in how we are conceiving, developing and evaluating them.

The Evidence Warehouse is a resource to help highlight evidence and answers to salient questions that patients, clinicians and researchers may have regarding health apps: What works? What doesnt work? Why?


The following articles advance our understanding of the importance of developing safe and effective mobile health apps.


Kumar S, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med. 2013 Aug;45(2):228-36.


ABSTRACT: Creative use of new mobile and wearable health information and sensing technologies (mHealth) has the potential to reduce the cost of health care and improve well-being in numerous ways. These applications are being developed in a variety of domains, but rigorous research is needed to examine the potential, as well as the challenges, of utilizing mobile technologies to improve health outcomes. Currently, evidence is sparse for the efficacy of mHealth. Although these technologies may be appealing and seemingly innocuous, research is needed to assess when, where, and for whom mHealth devices, apps, and systems are efficacious.

In order to outline an approach to evidence generation in the field of mHealth that would ensure research is conducted on a rigorous empirical and theoretic foundation, researchers gathered for the mHealth Evidence Workshop at NIH on August 16, 2011. The current paper presents the results of the workshop. Although the discussions at the meeting were cross-cutting, the areas covered can be categorized broadly into three areas: (1) evaluating assessments; (2) evaluating interventions; and (3) reshaping evidence generation using mHealth. This paper brings these concepts together to describe current evaluation standards, discuss future possibilities, and set a grand goal for the emerging field of mHealth research.

Wolf JA, et al. Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA Dermatol. 2013 Apr;149(4):422-6.


OBJECTIVES: To measure the performance of smartphone applications that evaluate photographs of skin lesions and provide the user with feedback about the likelihood of malignancy.

DESIGN: Case-control diagnostic accuracy study.

SETTING: Academic dermatology department.

PARTICIPANTS AND MATERIALS: Digital clinical images of pigmented cutaneous lesions (60 melanoma and 128 benign control lesions) with a histologic diagnosis rendered by a board-certified dermatopathologist, obtained before biopsy from patients undergoing lesion removal as a part of routine care.

MAIN OUTCOME MEASURES: Sensitivity, specificity, and positive and negative predictive values of 4 smartphone applications designed to aid nonclinician users in determining whether their skin lesion is benign or malignant.

RESULTS: Sensitivity of the 4 tested applications ranged from 6.8% to 98.1%; specificity, 30.4% to 93.7%; positive predictive value, 33.3% to 42.1%; and negative predictive value, 65.4% to 97.0%. The highest sensitivity for melanoma diagnosis was observed for an application that sends the image directly to a board-certified dermatologist for analysis; the lowest, for applications that use automated algorithms to analyze images.

CONCLUSIONS: The performance of smartphone applications in assessing melanoma risk is highly variable, and 3 of 4 smartphone applications incorrectly classified 30% or more of melanomas as unconcerning. Reliance on these applications, which are not subject to regulatory oversight, in lieu of medical consultation can delay the diagnosis of melanoma and harm users.

Bender JL, et al. A lot of action, but not in the right direction: systematic review and content analysis of smartphone applications for the prevention, detection, and management of cancer. J Med Internet Res. 2013 Dec 23;15(12):e287.


BACKGROUND: Mobile phones have become nearly ubiquitous, offering a promising means to deliver health interventions. However, little is known about smartphone applications (apps) for cancer.

OBJECTIVES: The purpose of this study was to characterize the purpose and content of cancer-focused smartphone apps available for use by the general public and the evidence on their utility or effectiveness.

METHODS: We conducted a systematic review of the official application stores for the four major smartphone platforms: iPhone, Android, Nokia, and BlackBerry. Apps were included in the review if they were focused on cancer and available for use by the general public. This was complemented by a systematic review of literature from MEDLINE, Embase, and the Cochrane Library to identify evaluations of cancer-related smartphone apps.

RESULTS: A total of 295 apps from the smartphone app stores met the inclusion criteria. The majority of apps targeted breast cancer (46.8%, 138/295) or cancer in general (28.5%, 84/295). The reported app purpose was predominantly to raise awareness about cancer (32.2%, 95/295) or to provide educational information about cancer (26.4%, 78/295), followed by apps to support fundraising efforts (12.9%, 38/295), assist in early detection (11.5%, 34/295), promote a charitable organization (10.2%, 30/295), support disease management (3.7%, 11/295), cancer prevention (2.0%, 6/295), or social support (1.0%, 3/295).

The majority of the apps did not describe their organizational affiliation (64.1%, 189/295). Apps affiliated with non-profit organizations were more likely to be free of cost (χ(2) 1=16.3, P<.001) and have a fundraising or awareness purpose (χ(2) 2=13.3, P=.001). The review of the health literature yielded 594 articles, none of which reported an evaluation of a cancer-focused smartphone application.

CONCLUSIONS: There are hundreds of cancer-focused apps with the potential to enhance efforts to promote behavior change, to monitor a host of symptoms and physiological indicators of disease, and to provide real-time supportive interventions, conveniently and at low cost. However, there is a lack of evidence on their utility, effectiveness, and safety. Future efforts should focus on improving and consolidating the evidence base into a whitelist for public consumption.

Marcano Belisario JS, et al. Smartphone and tablet self management apps for asthma. Cochrane Database Syst Rev. 2013 Nov 27;11:CD010013


BACKGROUND: Asthma is one of the most common long-term conditions worldwide, which places considerable pressure on patients, communities and health systems. The major international clinical guidelines now recommend the inclusion of self management programmes in the routine management of patients with asthma. These programmes have been associated with improved outcomes in patients with asthma. However, the implementation of self management programmes in clinical practice, and their uptake by patients, is still poor. Recent developments in mobile technology, such as smartphone and tablet computer apps, could help develop a platform for the delivery of self management interventions that are highly customisable, low-cost and easily accessible.

OBJECTIVES: To assess the effectiveness, cost-effectiveness and feasibility of using smartphone and tablet apps to facilitate the self management of individuals with asthma.

METHODS: We searched the Cochrane Airways Group Register (CAGR), the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, EMBASE, PsycINFO, CINAHL, Global Health Library, Compendex/Inspec/Referex, IEEEXplore, ACM Digital Library, CiteSeer(x) and CAB abstracts via Web of Knowledge. We also searched registers of current and ongoing trials and the grey literature. We checked the reference lists of all primary studies and review articles for additional references. We searched for studies published from 2000 onwards. The latest search was run in June 2013.

SELECTION CRITERIA: We included parallel randomized controlled trials (RCTs) that compared self management interventions for patients with clinician-diagnosed asthma delivered via smartphone apps to self management interventions delivered via traditional methods (e.g. paper-based asthma diaries).

DATA COLLECTION AND ANALYSIS: We used standard methods expected by the Cochrane Collaboration. Our primary outcomes were symptom scores; frequency of healthcare visits due to asthma exacerbations or complications and health-related quality of life.

RESULTS: We included two RCTs with a total of 408 participants. We found no cluster RCTs, controlled before and after studies or interrupted time series studies that met the inclusion criteria for this systematic review. Both RCTs evaluated the effect of a mobile phone-based asthma self management intervention on asthma control by comparing it to traditional, paper-based asthma self management. One study allowed participants to keep daily entries of their asthma symptoms, asthma medication usage, peak flow readings and peak flow variability on their mobile phone, from which their level of asthma control was calculated remotely and displayed together with the corresponding asthma self management recommendations. In the other study, participants recorded the same readings twice daily, and they received immediate self management feedback in the form of a three-colour traffic light display on their phones. Participants falling into the amber zone of their action plan twice, or into the red zone once, received a phone call from an asthma nurse who enquired about the reasons for their uncontrolled asthma.

We did not conduct a meta-analysis of the data extracted due to the considerable degree of heterogeneity between these studies. Instead we adopted a narrative synthesis approach. Overall, the results were inconclusive and we judged the evidence to have a GRADE rating of low quality because further evidence is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. In addition, there was not enough information in one of the included studies to assess the risk of bias for the majority of the domains. Although the other included study was methodologically rigorous, it was not possible to blind participants or personnel in the study. Moreover, there are concerns in both studies in relation to attrition bias and other sources of bias.

One study showed that the use of a smartphone app for the delivery of an asthma self management program had no statistically significant effect on asthma symptom scores (mean difference (MD) 0.01, 95% confidence interval (CI) -0.23 to 0.25), asthma-related quality of life (MD of mean scores 0.02, 95% CI -0.35 to 0.39), unscheduled visits to the emergency department (OR 7.20, 95% CI 0.37 to 140.76) or frequency of hospital admissions (odds ratio (OR) 3.07, 95% CI 0.32 to 29.83).

The other included study found that the use of a smartphone app resulted in higher asthma-related quality of life scores at six-month follow-up (MD 5.50, 95% CI 1.48 to 9.52 for the physical component score of the SF-12 questionnaire; MD 6.00, 95% CI 2.51 to 9.49 for the mental component score of the SF-12 questionnaire), improved lung function (PEFR) at four (MD 27.80, 95% CI 4.51 to 51.09), five (MD 31.40, 95% CI 8.51 to 54.29) and six months (MD 39.20, 95% CI 16.58 to 61.82), and reduced visits to the emergency department due to asthma-related complications (OR 0.20, 95% CI 0.04 to 0.99). Both studies failed to find any statistical differences in terms of adherence to the intervention and occurrence of other asthma-related complications.

CONCLUSIONS: The current evidence base is not sufficient to advise clinical practitioners, policy-makers and the general public with regards to the use of smartphone and tablet computer apps for the delivery of asthma self management programmes. In order to understand the efficacy of apps as standalone interventions, future research should attempt to minimise the differential clinical management of patients between control and intervention groups. Those studies evaluating apps as part of complex, multicomponent interventions, should attempt to tease out the relative contribution of each intervention component. Consideration of the theoretical constructs used to inform the development of the intervention would help to achieve this goal. Finally, researchers should also take into account: the role of ancillary components in moderating the observed effects, the seasonal nature of asthma and long-term adherence to self management practices.

Rosser BA and Eccleston C. Smartphone applications for pain management. J Telemed Telecare. 2011;17(6):308-12


ABSTRACT: Smartphone applications (or apps) are becoming increasingly popular. The lack of regulation or guidance for health-related apps means that the validity and reliability of their content is unknown. We have conducted a review of available apps relating to the generic condition of pain. The official application stores for five major smartphone platforms were searched: iPhone, Android, Blackberry, Nokia/Symbian and Windows Mobile. Apps were included if they reported a focus on pain education, management or relief, and were not solely aimed at health-care professionals (HCPs). A total of 111 apps met the inclusion criteria.

The majority of apps reviewed claimed some information provision or electronic manual component. Diary tracking of pain variables was also a common feature. There was a low level of stated HCP involvement in app development and content. Despite an increasing number of apps being released, the frequency of HCP involvement is not increasing. Pain apps appear to be able to promise pain relief without any concern for the effectiveness of the product, or for possible adverse effects of product use. In a population often desperate for a solution to distressing and debilitating pain conditions, there is considerable risk of individuals being misled.

O'Neill S and Brady RR. Colorectal smartphone apps: opportunities and risks. Colorectal Dis. 2012 Sep;14(9):e530-4


OBJECTIVES: The increased utilization of smartphones within the clinical environment together with connected applications (apps) provides opportunity for doctors, including coloproctologists, to integrate such technology into clinical practice. However, the reliability of unregulated medical apps has recently been called into question. Here, we review contemporary medical apps specifically themed towards colorectal diseases and assess levels of medical professional involvement in their design and content.

METHODS: The most popular smartphone app stores (iPhone, Android, Blackberry, Nokia, Windows and Samsung) were searched for colorectal disease themed apps, using the disease terms colorectal cancer, Crohn's disease, ulcerative colitis, diverticulitis, haemorrhoids, anal fissure, bowel incontinence and irritable bowel syndrome.

RESULTS: A total of 68 individual colorectal themed apps were identified, amongst which there were five duplicates. Only 29% of colorectal apps had had customer satisfaction ratings and 32% had named medical professional involvement in their development or content.

CONCLUSIONS: The benefits of apps are offset by lack of colorectal specification. There is little medical professional involvement in their design. Increased regulation is required to improve accountability of app content.

Visvanathan A et al. Smartphone apps in microbiology--is better regulation required? Clin Microbiol Infect. 2012 Jul;18(7):E218-20


ABSTRACT: Increasing diversity of available medical applications (apps) has led to their widespread use in healthcare delivery. However, app involvement in diagnosis and patient management has raised concerns, specifically regarding accuracy and reliability of content. Here, we report on the contemporary range of microbiology-themed apps and prevalence of medical professional involvement in app development. Of 94 microbiology-themed apps identified, only 34% had stated medical professional involvement. The lack of such involvement in app design is concerning and undermines consumers' ability to be informed regarding quality of content. We propose that increased regulatory measures are introduced to safeguard patient welfare.

Carter T, et al. Contemporary vascular smartphone medical applications. Ann Vasc Surg. 2013 Aug;27(6):804-9


BACKGROUND: Use of smartphones and medical mHealth applications (apps) within the clinical environment provides a potential means for delivering elements of vascular care. This article reviews the contemporary availability of apps specifically themed to major vascular diseases and the opportunities and concerns regarding their integration into practice.

METHODS: Smartphone apps relating to major vascular diseases were identified from the app stores for the 6 most popular smartphone platforms, including iPhone, Android, Blackberry, Nokia, Windows, and Samsung. Search terms included peripheral artery (arterial) disease, varicose veins, aortic aneurysm, carotid artery disease, amputation, ulcers, hyperhydrosis, thoracic outlet syndrome, vascular malformation, and lymphatic disorders.

RESULTS: Forty-nine vascular-themed apps were identified. Sixteen (33%) were free of charge. Fifteen apps (31%) had customer satisfaction ratings, but only 3 (6%) had greater than 100. Only 13 apps (27%) had documented medical professional involvement in their design or content.

CONCLUSIONS: The integration of apps into the delivery of care has the potential to benefit vascular health care workers and patients. However, high-quality apps designed by clinicians with vascular expertise are currently lacking and represent an area of concern in the mHealth market. Improvement in the quality and reliability of these apps will require the development of robust regulation.

Connor K, et al. Contemporary hernia smartphone applications (apps). Hernia. 2013 Jun 26


OBJECTIVES: Smartphone technology and downloadable applications (apps) have created an unprecedented opportunity for access to medical information and healthcare-related tools by clinicians and their patients. Here, we review the current smartphone apps in relation to hernias, one of the most common operations worldwide. This article presents an overview of apps relating to hernias and discusses content, the presence of medical professional involvement and commercial interests.

METHODS: The most widely used smartphone app online stores (Google Play, Apple, Nokia, Blackberry, Samsung and Windows) were searched for the following hernia-related terms: hernia, inguinal, femoral, umbilical, incisional and totally extraperitoneal. Those with no reference to hernia or hernia surgery were excluded.

RESULTS: 26 smartphone apps were identified. Only 9 (35 %) had named medical professional involvement in their design/content and only 10 (38 %) were reviewed by consumers. Commercial interests/links were evident in 96 % of the apps. One app used a validated mathematical algorithm to help counsel patients about post-operative pain.

CONCLUSIONS: There were a relatively small number of apps related to hernias in view of the worldwide frequency of hernia repair. This search identified many opportunities for the development of informative and validated evidence-based patient apps which can be recommended to patients by physicians. Greater regulation, transparency of commercial interests and involvement of medical professionals in the content and peer-review of healthcare-related apps is required.

Haffey F, et al. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf. 2013 Feb;36(2):111-7


BACKGROUND: Many medical professionals use smartphone applications (apps) on a daily basis to support clinical decision making. Opioid switching (conversion of one opioid to another at equianalgesic dose) is common in clinical practice and often challenging for doctors. Apps providing an opioid conversion tool can therefore be a useful resource. Despite rapid growth in the use of medical apps, the lack of robust regulation and peer review to ensure the accuracy and reliability of app content is currently an area of concern.

METHODS: We searched major online app stores for apps providing an opioid dose conversion tool. We assessed output variability between apps in the dose calculation of seven opioid switches, as well as assessing the level of professional medical involvement in the authorship, creation and design of the apps.

RESULTS: Of 23 different apps identified, more than half (n = 12; 52 %) had no stated medical professional involvement and only 11 (48 %) apps provided direct references to primary sources for their opioid conversion ratios. Conversion of 1 mg of oral morphine to oral codeine demonstrated the largest conversion output range (median 6.67 mg, range 3.333-12 mg). Conversion of 1 mg of oral morphine to methadone ranged from 0.05-0.67 mg, with only 44 % of methadone-converting apps (n = 4) commenting that the conversion ratio changes with magnitude of methadone dose. Overall, 35 % of apps (n = 8) did not warn the user about the standard practice of dose reduction when opioid switching. There was a statistically significant difference in the mean conversion output for hydromorphone (oral) between apps with and without medical professional involvement (0.2256 vs 0.2536; p = 0.0377).

CONCLUSIONS: There are significant concerns with regard to the reliability of information provided by apps offering opioid dose conversion, with lack of information regarding evidence-based content and peer review in many cases. It is crucial that better regulation of medical apps is instigated in order to ensure that patient safety is maintained.

Muessig KE, et al. Mobile phone applications for the care and prevention of HIV and other sexually transmitted diseases: a review. J Med Internet Res. 2013 Jan 4;15(1):e1


BACKGROUND: Mobile phone applications (apps) provide a new platform for delivering tailored human immunodeficiency virus (HIV) and sexually transmitted disease (STD) prevention and care.

OBJECTIVES: To identify and evaluate currently available mobile phone apps related to the prevention and care of HIV and other STDs.

METHODS: We searched the Apple iTunes and Android Google Play stores for HIV/STD-related apps, excluding apps that exclusively targeted industry, providers, and researchers. Each eligible app was downloaded, tested, and assessed for user ratings and functionality as well as 6 broad content areas of HIV prevention and care: HIV/STD disease knowledge, risk reduction/safer sex, condom promotion, HIV/STD testing information, resources for HIV-positive persons, and focus on key populations.

RESULTS: Search queries up to May 2012 identified 1937 apps. Of these, 55 unique apps met the inclusion criteria (12 for Android, 29 for iPhone, and 14 for both platforms). Among these apps, 71% provided disease information about HIV/STDs, 36% provided HIV/STD testing information or resources, 29% included information about condom use or assistance locating condoms, and 24% promoted safer sex. Only 6 apps (11%) covered all 4 of these prevention areas. Eight apps (15%) provided tools or resources specifically for HIV/STD positive persons. Ten apps included information for a range of sexual orientations, 9 apps appeared to be designed for racially/ethnically diverse audiences, and 15 apps featured interactive components. Apps were infrequently downloaded (median 100-500 downloads) and not highly rated (average customer rating 3.7 out of 5 stars).

CONCLUSIONS: Most available HIV/STD apps have failed to attract user attention and positive reviews. Public health practitioners should work with app developers to incorporate elements of evidence-based interventions for risk reduction and improve app inclusiveness and interactivity.

Huckvale K, et al. Apps for asthma self-management: a systematic assessment of content and tools. BMC Med. 2012 Nov 22;10:144


BACKGROUND: Apps have been enthusiastically adopted by the general public. They are increasingly recognized by policy-makers as a potential medium for supporting self-management of long-term conditions. We assessed the degree to which current smartphone and tablet apps for people with asthma offer content and tools of appropriate quality to support asthma self-management.

METHODS: We adapted systematic review methodology to the assessment of apps. We identified English-language asthma apps for all ages through a systematic search of official app stores. We systematically assessed app content using criteria derived from international guidelines and systematic review of strategies for asthma self-management. We covered three domains: comprehensiveness of asthma information, consistency of advice with evidence and compliance with health information best practice principles.

RESULTS: We identified 103 apps for asthma in English, of which 56 were sources of information about the condition and 47 provided tools for the management of asthma. No apps offered both types of functionality. Only three information apps approached our definition of comprehensiveness of information about asthma. No apps provided advice on lay management of acute asthma that included details of appropriate reliever medication use. In 32 of 72 instances, apps made unequivocal recommendations about strategies for asthma control or prophylaxis that were unsupported by current evidence. Although 90% of apps stated a clear purpose, compliance with other best practice principles for health information was variable. Contact details were located for 55%, funding source for 18% and confidentiality policy for 17%.

CONCLUSIONS: No apps for people with asthma combined reliable, comprehensive information about the condition with supportive tools for self-management. Healthcare professionals considering recommending apps to patients as part of asthma self-management should exercise caution, recognizing that some apps like calculators may be unsafe; that no current app will meet the need of every patient; and that ways of working must be adapted if apps are to be introduced, supported and sustained in routine care. Policy-makers need to consider the potential role for assurance mechanisms in relation to apps. There remains much to be done if apps are to find broad use in clinical practice; clinicians cannot recommend tools that are inaccurate, unsafe or lack an evidence base.