- Research article
- Open Access
Need-based care of multi-morbid patients – supporting general practitioners with algorithm-generated recommendations of healthcare services (telemedicine-project ATMoSPHÄRE)
BMC Family Practice volume 22, Article number: 198 (2021)
The patient-oriented and need-based care of multi-morbid patients with healthcare services and assistive products can be a highly complex task for the general practitioners (GPs). An algorithm-based digital recommendation system (DRS) for healthcare services was developed within the context of the telemedicine research project ATMoSPHÄRE. The plausibility of the DRS was tested and the results used to examine if, and to what degree, the DRS provides useful assistance to GPs.
The plausibility of the recommendations of the DRS were tested with the Delphi procedure (n = 8) and Interviews (n = 4) in collaboration with the GPs. They proposed services and assistive products they considered appropriate for two multi-morbid patients. Furthermore, GPs had to report whether, and to what degree they deemed the algorithm-generated recommendations appropriate. Significant quantitative differences between the GPs’, and the algorithm-generated, recommendations were evaluated with paired-samples-Wilcoxon-test.
The first Delphi round revealed a high variability regarding the amount and character of services recommended by the physicians (1 to 10 recommendations, mean = 5.6, sd = 2.8). These professional recommendations converged after consideration of the algorithm-generated recommendations. The number of algorithm-generated recommendations which were judged as appropriate ranged between 7 and 17 of a total of 20 (mean = 11.9, sd = 2.5). The interviews revealed that the additional algorithm-generated recommendations which were judged appropriate contained mainly social care services.
The DRS provides GPs with additional appropriate recommendations for the need-based care of patients, which may not have been previously considered. It can therefore be assessed as a helpful complement in the primary care of multi-morbid patients.
The increasingly growing quantity and complexity of available data has led to a renewed attention towards algorithms in IT Solutions and Systems . Recommender systems are, depending on complexity and heterogeneity of necessary information, based on more or less complex algorithms. They aim to present selected, personalized content and offers to the user that could be of particular interest to them. They differ in the way they analyze data and their data sources. One way to generate recommendations is to compare the user data against the whole user database, identifying matches between the user and other users with similar behavior . These recommender systems are primarily used by online eCommerce services such as Amazon and Netflix [2, 3].
In the context of healthcare, personal patient data and medical expertise serve as the main foundation for a patient-focused therapy and care solution. The progressing digitalization of medicine and healthcare means that much of this information is also available in digital formats. Using this data for assistance or recommender systems in clinical work should improve patient care. Recommender systems are increasingly emerging in the medical field and eHealth applications, for instance as algorithm-driven health service recommenders , clinical decision support systems  or for shared decision-making . Recommender systems are now used to support the management of various diseases, such as heart disease  or diabetes  but also for example in smoking cessation .
The rising prevalence of chronic diseases amplifies the challenge that multimorbidity (multimorbidity defined as the concurrent presence of two or more chronic diseases) poses in the primary care of older patients . Multimorbid patients constitute a very diverse group regarding disease combination, severity and treatment. The GP has to consider a multitude of information, often acquired in a long-lasting patient-doctor relationship.
Central treating targets of patient-centered care could be: involvement of the multimorbid patient in the prioritization of treatment targets (shared decision-making) , enabling the patient to remain at home, the preservation of patient functions, and the avoidance of hospital admissions. A multitude of options for treatment, consultation, and referral of support services exists to pursue these targets in primary care. The range of services aimed at improving and preserving independent living, in particular, continues to increase, driven by higher demand and new technological possibilities of recent years [11, 12].
This evolving diversity could be made available to the GPs by using patient-data-driven recommender systems to support the consultation in a way that is demand-oriented and also considers available care.
A digital recommendation system (DRS) was developed in the context of the joint telemedicine project ATMoSPHÄRE, employing an algorithm to match patients, according to their data, with appropriate health care services. Based on a plausibility check the appropriateness of the DRS was evaluated. The results were used to investigate the question of whether, and to what degree, the DRS can support the treatment of multimorbid patients in primary care.
The telemedicine project ATMoSPHÄRE started 2015 in Saxony and developed, employed, and evaluated a technology-based information and communication platform and a complementary home-based telemonitoring application. The aim of these tools was to improve the primary care of older, multi-morbid patients and therefore preserve and extend their ability to live independently. The telemedicine platform contained, in addition to the data provided by the telemonitoring application, information about socio-demographics, diagnoses, and therapy, as well as the results of various geriatric assessments. Detailed information about the study, including patient recruitment, study design, and additional information about the telemonitoring application developed for the patients, has been published elsewhere [13, 14].
Digital recommendation system
The development and implementation of a digital recommendation system for external healthcare services by vital.services GmbH was a pivotal part in the second stage of ATMoSPHÄRE.
The DRS was intended to offer need-based, individual recommendations for medical and social care services to the study GPs, thereby assisting them in the planning of treatment.
Embedded into the telemedicine platform, the DRS used an algorithm to compare functional limitation of a patient with the treatment targets of a healthcare service (e.g. targets of occupational therapy), and determine how closely they matched. The DRS ranked all services according to how well they matched, and presented the closest matching services to the study GPs as recommendations on the telemedicine platform (Fig. 1).
The functional limitations were gathered using mostly commonly-employed geriatric assessments (Instrumental Activities of Daily Life (IADL) , Mini Mental State Examination (MMSE) , clock test , Timed Up-and-Go , ANGELINA , Geriatric Depression Scale (GDS) ). The treatment targets of services were taken or derived from different catalogues [21, 22]. Comparing functional limitations and treatment targets required to classify them within the framework of the International Classification of functioning, disability and health (ICF) . Apart from providing a consistent and distinct vocabulary, the ICF possesses two additional properties utilized by the the DRS. Firstly, the mono-hierarchical structure allows definition of a proximity between categories. Secondly, the ICF-qualifier ratings signify the severity of disability within the corresponding category, which the DRS interprets as an indicator of need. The higher the severity of a functional limitation, the higher the need of a patient for a service which aims to alleviate that limitation.
Provided the limitations in functioning of a patient are classified within the ICF, each with the corresponding qualifier rating, and a set of healthcare services exists where the treatment targets of each service are converted into ICF-categories, a ranking of these healthcare services can be established. The treatment targets of a service or intervention are a set of ICF-categories representing the areas of functioning the service aims to support or improve. A service is considered more fitting the higher the qualifier rating of the function is, and the closer the function category and treatment target category are. An ICF category represents a specific function. The ICF qualifier indicates how severely the function is impaired. The qualifier ranges from 0 - meaning no impairment to 4 - indicating complete loss of function.
The functionality is shown schematically in Fig. 1 using the example of an impaired patient function indicated by the “Timed-Up-and-Go-Test” (relevant mobility impairment in need of clarification). The algorithm compares the ICF-coded assessment result of an individual patient to the ICF- categories representing the treatment targets of “walking frame”. For instance, the assessment result d4500.3 represents a severe impairment of the ability to walk short distances, this matches closely the treatment target d450 – Walking of the walking frame indicating this service could be appropriate. The closer the match and the higher the qualifier of the patient the higher the services will be ranked.
This publication bases on the results of the plausibility check of the DRS (Version 3.1.3) carried out by the Department of General Practice of the TU Dresden, which examined the adequacy of the DRS regarding its individually-compiled service recommendations for multi-morbid patients. This article is not intended to show the results of the entire plausibility check but rather presents the results of the first Delphi round and the interviews which entail the original answers and recommendations of the study GPs.
The operationalization of the adequacy of the DRS was achieved by a comparison with a determined reference value. The number and character of algorithm-generated recommendations were compared to the reference value regarding matching, missing, and additional fitting recommendations.
The reference value should represent the shared knowledge of a certain group, e.g. experts in a specific area . In this context, the reference value is the set of the study GPs’ recommendations deemed appropriate for the treatment of a specific multimorbid patient.
The plausibility check followed a mixed-method approach and consisted of a) a two-staged Delphi procedure, evaluating the recommendations for two case vignettes (cv), and b) interviews with ATMoSPHÄRE study GPs, evaluating recommendations for selected study patients.
Based on a previous analysis of the most common supply needs of the study cohort, two profiles for multimorbidity were created. Patients corresponding to profile 1 all had indications for polypharmacy and were additionally diagnosed with type 2 diabetes mellitus and hypertension. Profile 2 patients showed irregularities in dementia-related assessments (MMSE or clock-test), or were already diagnosed with dementia. Using these two profiles, two corresponding study patients were selected and then utilized as anonymous case vignettes during the Delphi procedure. The data of these patients stand vicariously for the ailments and treatment issues patients of these profiles present during consultation hour. In the same way, two study patients were selected to be evaluated by study GPs for the evaluation via interviews.
The Delphi procedure was carried out in writing and consisted of two survey rounds. In both rounds the study GPs received a two-part questionnaire. The original GP- recommendations for the case vignettes (Part 1) as well as their approval or disapproval for the DRS-recommendations (Part 2) were collected in the first Delphi round. In the second round of the Delphi process the GPs were confronted with their own recommendations/answers and the recommendations of the other participating GPs and given the opportunity to reconsider decisions made in the 1st round (Fig. 2).
The purpose of the second round was to establish a consensus among the GPs which in turn was used to determine the reference value needed to evaluate the DRS-recommendations. This round is therefore the basis of the plausibility check but is not presented in this article.
In the first part of the questionnaire two case vignettes were introduced, including socio-demographic context, diagnoses, medication, and results of the geriatric assessments (see Fact sheet in Tables 1 and 2). The study GPs recommended healthcare interventions and services which they considered helpful for the case vignettes at hand. Part 1 was therefore used to determine the reference value. In Part 2 the recommendations of the DRS for the two case vignettes were presented to the study GPs. They were asked to judge which recommendations they considered as appropriate or inappropriate (see Additional file 1).
Additionally, two actual study patients (not case vignettes) representing the 2 profiles were discussed with their GP in the interviews in the same manner (Part 1 + Part 2). Furthermore, the study GPs were interviewed about the DRS in general. The interview questions for the plausibility check were developed based on the SPSS principle according to Helfferich . The interviews were conducted by a trained researcher at GPs’ office.
The statistical analysis (with SPSS Version 25) of the quantitative data consisted, in addition to data description, of a comparison of means of the number of recommendations (non-parametric Wilcoxon Rank Sum Test for paired samples) between GPs and DRS. To determine the relevance of the results, the effect size according to Cohen was calculated. Effect sizes with values of d ≥ 0.8 are considered strong effects . A significance level α equal to 5% was defined.
Of the 14 enlisted study GPs, 8 participated in the Delphi procedure (m = 2, f = 6; age (n = 6): mean = 50.7, sd = 7.8). They were working within the primary care system in the state of Saxony, with a minimum of 10 years of professional experience. Four of the 8 study GPs participated in the interviews, as they care for the study patients that represent the patient profiles.
The study GPs indicated between three and ten (mean = 6, sd ± 2.9) recommendations for case vignette no. 1 (Table 1), and between one and nine (mean = 5.1; sd ± 2.9) for case vignette no. 2, respectively (Table 2). In comparison, the average number of appropriately rated algorithm-generated recommendations was significantly higher (asymptotic Wilcoxon-Test for both case vignettes: z = − 2.53, p = 0.011, n = 8), and roughly twice as many (case vignette no.1: mean = 11.9, sd ± 2.6; case vignette no.2: mean = 12, sd ± 2.6). The effect size according to Cohen was above 2.0 for both case vignettes (Fig. 3).
Examining the individual GP recommendations shows that only the first three recommendations for case vignette no.1 were made by a majority of physicians, meaning more than 4 of the 8 participating study GPs (Table 1). Likewise, only three recommendations were proposed by a majority for case vignette no. 2 (Table 2). After presentation of the algorithm-generated recommendations, the judgements converged (case vignette no.1: majority approval of 10 recommendations; case vignette no.2: majority approval of 12 recommendations) (Additional file 1).
A similar trend was found on conducting the interviews, where Part 1 consisted of the question: “Which prescriptions, healthcare interventions or services would you propose to your patient x?” Here, too, a significantly higher number of algorithm-generated recommendations were judged appropriate than previously listed by the study GPs for the patient (asymptotic Wilcoxon-Test: z = − 2.37, p = 0.018, n = 8). The effect size for the recommendations in the interviews was d = 1.76, which means a strong effect (Fig. 3).
Comparing the character of the recommendation categories shows that the DRS-recommendations additionally judged appropriate can be mainly assigned to social care (e.g. emergency-call service, pick-up and return services) and assistive products (e.g. pill dispensers). Occasional GP-made recommendations not included in the DRS pertain to special nursing services, or special assistive products, respectively.
All four interviewees confirmed in principle that the DRS is of added value to the GPs and the patient care. Requesting for specific assessment of the DRS prompted diverse responses. On one hand, the DRS was considered a helpful addition and reminder: “[...] you might ask yourself: “Useful or not?”, but as a check or suggestion, that I do like.[...] I don’t always consider foot care if patients don’t have diabetes. That surely is something.” (A1, f, 43) or: “Well, it is useful to get a reminder. That’s alright, I guess.” (A2, m, 44). On the other hand, the vagueness due to a lack of consideration of the patients’ priorities was criticized: “It’s tough. Because I believe a lot of hard and soft data is missing for the digital recommendation system. The patient wishes for instance, the prioritisation, what is most important to the patient at that time,[...].” (A3, f, 48).
The results of the first Delphi round representing the original answers of the study GPs show that despite having the same information available during the Delphi procedure, the study GPs’ recommendations varied heavily with respect to number and character. This points to the phenomenon that identical patient data can be interpreted very differently, subsequently leading to diverse expectations of patient needs, or that GPs may derive very different priorities respectively. Previous experience and methods, but also a lack of knowledge about (new) healthcare interventions and services, might explain the differing responses from the GPs.
In Part 2 the study GPs judged the DRS-recommendations according to their appropriateness and suitability for the case vignettes/actual study patients.
The study GPs judged a higher number of DRS recommendations appropriate, than they themselves made initially, for the case vignettes in the Delphi procedure, and study patients in the interviews, alike. The study GPs considered the algorithm-generated recommendations as a helpful complement or reminder of services. The recommendations reminded the study GPs of services which are rarely considered, or even forgotten, in day-to-day business. The recommendations deemed additionally appropriate pertained mainly to social care and assistive products.
However, the DRS failed to generate recommendations of some services which the study GPs considered necessary (e.g. “assistive products for personal hygiene” at cv no.1). This might be because the DRS could not match the patients’ limitation of functioning with the treatment target of the services in question, or the services were not available to the system because no reliable source was found which either had the treatment targets of these services already linked to ICF-categories or allowed for verifiable linking to the ICF.
The analysis shows potential for further optimization: the addition of more data for the DRS to evaluate, specifically the wishes and priorities of the patient, which cannot be derived from the geriatric assessments. Patient priorities should be considered in the therapeutic goals and in decisionmaking. Furthermore, incorporating the patients’ priorities increases trust and adherence .
Strengths and limitations
As shown in other studies regarding the employment of assessments, geriatric assessments of multimorbid patients are only the basis for shared decision-making of therapeutic goals and healthcare decisions [28,29,30].
The enduring participation of GPs during the plausibility check testifies to the interest of physicians in this subject. Due to the small sample size and restriction to patient profiles in the plausibility check, the results cannot be generalised. A heightened interest into telehealth applications by the participating GPs can be assumed, leading to a likely selection bias. This might have intensified the positive judgement of a digital tool.
The “patient-data-driven DRS of healthcare services” was developed as an algorithm-based recommender system in the context of the telemedicine project ATMoSPHÄRE, to provide GPs with patient-related and need-based proposals of healthcare services, with the goal to support them in treatment and healthcare decisions. The results of this study show that the DRS can support GPs treating multimorbid patients, by generating additional, potentially-helpful suggestions, which are unknown to, or not considered by the GP, but which may aid the patient in maintaining their functions and improve their quality of life.
A DRS can replace neither the doctor-patient-relationship, nor the joint prioritisation of therapeutic goals. The additional recommendations may, however, facilitate the dialogue between doctor and patient, and support patient-centered need-based care, which takes current healthcare services into account.
The identified potential for optimisation can be used to develop the system further. For instance, the ability to prioritise individual treatment targets according to the wishes of the patient as well as the addition of more services into the DRS-database would be beneficial. Regardless of the match-making algorithm, the system could benefit from a feature showing relevant healthcare service providers in proximity of the patient. Providing this information to the patient, by using the telemedicine application, could facilitate therapy planning.
Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author PB on reasonable request.
Digital recommendation system
Geriatric Depression Scale
Instrumental Activities of Daily Life
International Classification of functioning, disability and health
Mini Mental State Examination
Wu X, Zhu X, Wu G, Ding W. Data mining with big data. IEEE Trans Knowl Data Eng. 2014;26(1):97–107.
Melville P, Sindhwani V. Recommender systems. In: Sammut C, Webb GI, editors. Encyclopedia of machine learning and data mining. Boston: Springer; 2017. p. 1056–66.
Burke R, Felfering A, Göker MH. Recommender systems: an overview. IA Mag. 2011;32:13–8.
Graesser F, Beckert S, Kuster D, Schmitt J, Abraham S, Malberg H, et al. Therapy decision support based on recommender system methods. J Healthc Eng. 2017;2017:8659460.
Danial-Saad A, Kuflik T, Weiss PL, Schreuer N. Effectiveness of a clinical decision support system for pointing device prescription. Am J Occup Ther. 2015;69(2):6902280010p1–7.
Shaoibi A, Neelon B, Lenert LA. Shared decision making: from decision science to data science. Med Decis Making. 2020;40(3):254–65.
Mustaqeem A, Anwar SM, Khan AR, Majid M. A statistical analysis based recommender model for heart disease patients. Int J Med Inform. 2017;108:134–45.
Norouzi S, Kamel Ghalibaf A, Sistani S, Banazadeh V, Keykhaei F, Zareishargh P, et al. A mobile application for managing diabetic patients’ nutrition: a food recommender system. Arch Iran Med. 2018;21(10):466–72.
Hors-Fraile S, Schneider F, Fernandez-Luque L, Luna-Perejon F, Civit A, Spachos D, et al. Tailoring motivational health messages for smoking cessation using an mHealth recommender system integrated with an electronic health record: a study protocol. BMC Public Health. 2018;18(1):698.
World Health Organization. Multimorbidity: technical series on safer primary care. Geneva: World Health Organisation; 2016. https://apps.who.int/iris/handle/10665/252275.
bcc Research. Assistive devices for special needs: technologies and global markets. 2018.
Research and Markets. World elderly and disabled assistive devices market - opportunities and forecasts, 2017–2023. 2019.
Lang C, Scheibe M, Voigt K, Hübsch G, Mocke L, Schmitt J, et al. Reasons for non-acceptance and non-use of a home telemonitoring application by multimorbid patients aged 65 years and over. Z Evid Fortbild Qual Gesundhwes. 2019;141:76–88.
Lang C, Voigt K, Neumann R, Bergmann A, Holthoff-Detto V. Adherence and acceptance of a home-based telemonitoring application used by multi-morbid patients aged 65 years and older. J Telemed Telecare. 2020:1357633X20901400. Online ahead of print.
Lawton MP, Brody EM. Assessment of older people: self-maintaining and instrumental activities of daily living. The Gerontologist. 1969;9(3):179–86.
Folstein MF, Folstein SE, McHugh PR. “Mini-mental state”: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12(3):189–98.
Shulman KI, Pushkar Gold D, Cohen CA, Zucchero CA. Clock-drawing and dementia in the community: a longitudinal study. Int J Geriatr Psychiatry. 1993;8(6):487–96.
Podsiadlo D, Richardson S. The timed “up & go”: a test of basic functional mobility for frail elderly persons. J Am Geriatr Soc. 1991;39(2):142–8.
Leipziger Gesundheitsnetz, GeriNet Leipzig. Angelina: Identifikation geriatrischer Patienten. 2014.
Yesavage JA, Brink TL, Rose TL, Lum O, Huang V, Adey M, et al. Development and validation of a geriatric depression screening scale: a preliminary report. J Psychiatr Res. 1982;17(1):37–49.
Deutscher Verband der Ergotherapeuten e.V. Indikationskatalog Ergotherapie. Idstein: Schulz-Kirchner Verlag GmbH; 2017.
GBA. Zweiter Teil Zuordnung der Heilmittel zu Indikationen (Heilmittel-Katalog) Zuordnung der Heilmittel zu Indikationen nach § 92 Abs. 6 Satz 1 Nr. 2 SGB V.
Heerkens Y, Bougie T, Claus E. The use of the ICF in the process of supplying assistive products: discussion paper based on the experience using a general Dutch prescription guideline. Prosthetics Orthot Int. 2011;35(3):310–7.
Böhnert M, Reszek P. Linguistisch-philosophische Untersuchung zu Plausibilität: über kommunikative Grundmuster bei der Entstehung von wissenschaftlichen Texten. 2015. In: Engelschalt J, Maibaum A, editors. Auf der Suche nach den Tatsachen: Proceedings der 1 Tagung des Nachwuchsnetzwerks “INSIST”, 22–23 Oktober 2014. 2015. p. 40–67.
Helfferich C. Die Qualität qualitativer Daten. Wiesbaden: Verlag für Sozialwissenschaften; 2011.
Cohen J. A power primer. Psychol Bull. 1992;112(1):155–9.
Dehn-Hindenberg A. Quality indicators from the patient’s viewpoint. Key to therapeutic success: communication, mutual trust, patient-centeredness. Gesundheitsökonomie & Qualitätsmanagement. 2008;13(05):298–303.
Voigt I, Wrede J, Diederichs-Egidi H, Dierks M-L, Hummers-Pradier E, Junius-Walker U. PrefCheck: patient-centered treatment planning with older multimorbid patients. Z Gerontol Geriatr. 2010;43(5):303–9.
Theile G, Muller CA. Multimorbid general practice patients - what's really important? Praxis (Bern 1994). 2012;101(25):1621–6.
Afshar K, Bokhof B, Wiese B, Dierks M-L, Junius-Walker U. Patients’ need for consultation after a geriatric assessment in family practice. Z Gerontol Geriatr. 2016;49(7):632–8.
We thank our consortium partners of the ATMoSPHÄRE project for their constructive cooperation. Special thanks go to the study GPs who took part in the plausibility check. The authors also want to thank all study patients for their participation in the study, which in some cases spans over several years.
The development of the DRS and its plausibility check took place within the ATMoSHÄRE project. This project was funded by the German Federal Ministry of Education and Research (No. 13GW0075F). The funder had no influence on the conception, conduction, analysis or interpretation of the plausibility check. Open Access Funding by the Publication Fund of the TU Dresden. Open Access funding enabled and organized by Projekt DEAL.
Ethics approval and consent to participate
The study was approved by the ethics committee at the Technische Universität Dresden (approval number 1012016). Written consents were obtained from the participants.
Consent for publication
All authors confirm that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
DRS-recommendations for cv no.1 judged appropriate by study GPs (Data from 1st Delphi round Part 2). Appendix 2. DRS-recommendations for cv no.2 judged appropriate by study GPs (Data from 1st Delphi round Part 2).
About this article
Cite this article
Borchers, P., Piller, S., Böhme, M. et al. Need-based care of multi-morbid patients – supporting general practitioners with algorithm-generated recommendations of healthcare services (telemedicine-project ATMoSPHÄRE). BMC Fam Pract 22, 198 (2021). https://doi.org/10.1186/s12875-021-01537-2
- Recommender system
- Primary care
- Decision support system