An intelligent system for improving adherence to guidelines on acute stroke

Abstract

OBJECTIVES: A timely, accurate assessment and decision-making process is essential for the diagnosis and treatment of the acute stroke, which is the world’s third leading cause of death. This process is often performed using the traditional method that increases the complexity, duration, and medical errors. The present study aimed to design and evaluate an intelligent system for improving adherence to the guidelines on the assessment and treatment of acute stroke patients.

METHODS: Decision-making rules and data elements were used to predict the severity and to treat patients according to the specialists’ opinions and guidelines. A system was then developed based on the intelligent decision-making algorithms. The system was finally evaluated by measuring the accuracy, sensitivity, specificity, applicability, performance, esthetics, information quality, and completeness and rates of medical errors. The segmented regression model was used to evaluate the effect of systems on the level and the trend of guideline adherence for the assessment and treatment of acute stroke.

RESULTS: Fifty-three data elements were identified and used in the data collection and comprehensive decision-making rules. The rules were organized in a decision tree. In our analysis, 150 patients were included. The system accuracy was 98.30%. Evaluation results indicated an error rate of 1.69% by traditional methods. Documentation quality (completeness) increased from 78.66% to 100%. The average score of system quality was 4.60 indicating an acceptable range. After the system intervention, the mean of the adherence to the guideline significantly increased from 65% to 99.5% (P < 0.0008).

CONCLUSION: The designed system was accurate and can improve adherence to the guideline for the severity assessment and the determination of a therapeutic trend for acute stroke patients. It leads to physicians’ empowerment, significantly reduces medical errors, and improves the documentation quality.

Introduction

Acute stroke is rapidly developing clinical signs of focal (or global) disturbance of cerebral function, lasting for more than 24 h or leading to death without any apparent cause, unlike vascular origins.[1] This disease affects all ages, but its incidence rate [2] increases by age. According to the statistics, acute stroke affects more than 15 million people in the world every year. It is also known as the third cause of death.[3,4] It is vital to utilize a quick, timely and an accurate method for the assessment and treatment of acute stroke as the wrong and time-consuming decision-making leads to irreparable complications for patients.[5] Assessment and treatment guidelines for acute stroke diseases aim to improve the outcomes and cost-effectiveness. However, it remains a challenge to effectively integrate these guidelines into clinical practice because medical providers do not have adequate sources, time, and mental focus for the treatment to provide accurate healthcare services, and also, they are often faced with incomplete data, leading to increasingly misdiagnosis and medical errors.[6] Using modern technologies, such as intelligent systems, can be an effective way to solve this challenge.[7]

Intelligent systems were defined as the system that incorporates intelligence into applications being handled by machines. Intelligent systems also perform complex automated tasks that are not possible by the traditional computing paradigm. Rule-based reasoning systems are the form of artificial intelligence system.[8] This system uses rules as the knowledge representation for knowledge coded into the system. The definitions of the rule-based systems depend almost entirely on expert systems, which are a system that mimics the reasoning of a human expert in solving a knowledge-intensive problem. Instead of representing knowledge in a declarative, static way as a collection of facts that are valid, rule-based system represents knowledge in terms of a set of rules that tells what to do in various circumstances or what to infer.[9]

These systems, if properly designed, play an important role in collecting useful information such as initial data, documents, personal knowledge, and business models to solve problems and subsequently allow decision-makers to quickly administrate a huge volume of information computing and processing. They can thus be used as a solution for complex issues and emergency situations that often require quick and accurate responses.[10] Our study was conducted with the aim to design and evaluate an intelligent system for increasing adherence to the guidelines on assessment and treatment of acute stroke to reduce error rates and increase the diagnostic accuracy in emergency situations.

Material and Methods

Information requirement engineering

This process is considered one of the most critical aspects of constructing an intelligent system because during this process is determined what is to be designed.[11] We first observed the acute stroke assessment and treatment workflow in some emergency departments (EDs). Then, we conducted unstructured interviews with four clinicians (emergency medicine specialist and neurologist) to examine the current efficiency of the existing system and determined the major weak and strong points. The clinicians were asked about their perceptions, opinions, beliefs, and attitudes toward the shortcomings of the paper-based workflow. The potential strong points of implementing an intelligent system were explained to them. The Ethics Committee of Tabriz University of Medical Sciences approved the procedures of the study.

Intelligent system was developed in three phases consisting of descriptive, developmental, and evaluation data [Figure 1]. To identify and determine the data elements for patient assessment and determination of acute stroke therapeutic trends in the descriptive phase, guidelines and published resources were first searched and collected on active global websites in the field of acute stroke. Collected resources were then provided for four emergency medicine physicians and neurologists as a checklist. They reviewed the checklist and selected and approved important resources. In the next step, data elements were extracted from verified resources. To determine a decision-making process (decision model), a focus group meeting was held with four physicians (emergency medicine specialist and neurologist) and three technical specialists as the members of the intelligent system team who were responsible for system implementation in a clinical environment. During this meeting, team experts carried out a focus group discussion that was facilitated by a health information technology expert and used the verified data elements to create decision-making rules.

System design and development

In the design and development phase, the system was developed in an agile methodology based on the extracted rules and knowledge in the previous phase.

In the design phase, the system architecture was proposed based on the existing technologies for developing diagnostic and treatment systems. System features and axes were then determined for requirements analysis to integrate intelligent tools and visualize knowledge. The system was finally designed in a comprehensive decision tree. In the present study, the rule-based reasoning determined related decision-making rules based on the representation of the domain knowledge that was available in guidelines, studies, and expert’ opinions using the “if-then” format. Other decision-making points were modeled as qualitative or quantitative rules. Figure 2 shows the knowledge representation process. Besides, some requirements were explained and modeled by selected unified modeling language diagrams such as the use case.

In the development phase, the user interface and database layers of the system were designed using Android Studio, Android SQLLite, and Java programming language in a monolingual (English) format. The database contained four tables including physician demographic data, patient information, acute stroke severity, and therapeutic trend as well as reasons for proposed therapeutic procedures. The intelligent system was an android mobile application.

System function

When the system was running, it asked a specialist physician to create a user profile. The system then asked the specialist physician to enter the patient’s code. In the next step, the system evaluated and determined the severity of the patient’s acute stroke based on the items selected by the specialist physician. After the severity of the patient’s acute stroke is determined, the system suggested treatment methods based on the decision-making rules (guidelines) in its rule base and the conditions selected by the specialist physician. After the process was completed, the system showed a form containing the details of the relevant specialist, the patient’s code, the severity of the acute stroke, the chosen treatment, and the reason for choosing that treatment and stored in the database. The system had features such as profile editing, reset, search, settings, ability to go back to the previous step, status bar, and help and explain how to use in each step.

Data collection and system evaluation

In the evaluation phase, the application was installed on the emergency medicine specialist (resident of emergency medicine) smartphone or tablet. The emergency medicine specialist (resident of emergency medicine) was trained on using the application. Then, the designed system was exposed to users and evaluated using the mobile application rating scale (MARS) tool, which provided comprehensive coverage of user experience,[12] as well as accuracy, sensitivity, specificity, physicians’ errors rates, guideline adherence, and documentation quality (completeness).

All data were collected from the patients who had visited the ED of Imam Reza Hospital (Tabriz-Iran) from March 2018 to August 2018. Data were also simultaneously collected by physicians (residents of emergency medicine) on paper forms and through system users on an intelligent system. During the data collection period, data from patients whose primary diagnosis was an acute stroke were included in the study, and data from other patients were excluded. Further, we included residents of emergency medicine with more than 3 years’ experience in the ED and interested to collaborate in the study, and others were excluded from the study. The system was evaluated based on the collected dataset, which included 150 items, as well as the MARS questionnaire that was completed by relevant experts.

A confusion matrix was used to measure the accurate prediction of the classification model and physicians’ (residents of emergency medicine) error rates using a paper-based method. The measurement was aimed to determine the accuracy, sensitivity, specificity, and error rates as presented in the following equations:

Sensitivity = (TP / TP + FN)%
Specificity = (TN / TN + FP)%
Accuracy = (TP + TN / Totall)%
Error rate = (FP + FN / Totall)%

Where TP refers to a true-positive rate; TN is a true-negative rate; FP is a false-positive rate, and FN is a false-negative rate.

The adherence percentages were assessed with the time series data. The segmented regression model was used to evaluate the effect of the system on the level and trend of guideline adherence in the severity assessment and determination of a therapeutic trend of acute stroke. Guideline adherence was determined by the concordance between guideline-recommended therapy and prescribed treatment. The time-series data are the most efficient, quasi-experimental method for evaluating the longitudinal effects of such time-limited interventions. Segmented regression analysis of time series data helps us to measure, in statistical terms, how much an intervention changed an outcome of interest, immediately and over time; instantly or with delay; transiently or long-term; and whether variables other than the intervention could explain the change.[13]

Two assessors reviewed paper forms and evaluated the number and distribution of completeness or incompleteness of records to evaluate the completeness of documentation. A technical assessor (query) did the same for registered cases in the system. Incompleteness means that a record has one or more missing data items in every form. The following equations were used to calculate the record completeness measure:

Completeness = (Total complete records / Total records) %,
Incompleteness = ((1− completeness)×100)%

Descriptive statistics were used to describe patient and expert attributes. A segmented regression analysis was used for determining the change in level and trend of adherence after using the system. The Durbin–Watson test was used to test the autocorrelation in the regression model. The statistical analysis was performed in IBM Corp. Released 2016. IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp. A P < 0.05 was considered statistically significant. Figure 1 shows the overview of the study methodology.

Results

Descriptive data analysis

Two members of the focus group were emergency medicine specialist; two members were neurologist, and three members were technical expert. Four members of the participants worked in the hospital, and three members were employed at the university. Four members of the participants also had more than 5 years of job experience. The mean age of the visited patients was 56 years. 98 people (65.33%) out of 150 patients with acute stroke were male and 52 (34.67%) were female. In addition, most patients were illiterate and married. Table 1 presents the characteristics of focus group members and patients.

Fifty-three data elements were identified for acute stroke severity assessment and therapeutic trend determination. To assess the severity of the acute stroke, the National Institutes of Health Stroke Scale (NIHSS) scores were selected and confirmed from all the items. NIHSS is a reliable, accurate, and sensitive instrument for measuring the severity of the stroke. NIHSS needs less equipment for a stroke patient to be provided quickly. It is also free to support doctors and conveniently available. Some evidence indicates that the NIHSS is sensitive over time toward recognizing important clinical changes.[14] Therapeutic trends include traditional therapies, fibrinolytic therapy, and thrombectomy. Considering criteria such as indications and contraindications for fibrinolytic therapy , and indications and contraindications for thrombectomy were also selected and approved for the treatment of acute stroke. Results of the used resources and data elements for creating the decision-making rules and decision tree were presented in Appendices 1-12.

Rule‐based reasoning results

The extracted rules, including 43 rule-based reasoning, were created in the rule-based engine and were used for the assessment and treatment of acute stroke patients. Figure 2 shows the knowledge representation process as a decision tree.

The present study recommended an appropriate architecture. The architecture consists of seven main components: user- and case-specific data, database, rule base, inference engine, explanation system, rule base editor, and the user interface. The provided system architecture is presented in Appendices 1-12.

Figure 3 (Part A) shows a section of the designed graphic user interface for the NIHSS calculation and an acute stroke severity assessment module. In this module, data elements related to the acute stroke severity assessment were entered in 15 categories, including the level of consciousness which itself has three parts, A, B, and C, best Gaze, visual, facial palsy, the motor arm which has two parts, right and left, motor leg which has two parts, right and left, limb ataxia, sensory, best language, dysarthria, extinction, and inattention.

In the acute stroke therapeutic trend determination module [Figure 3 (Part B)], the therapeutic trend was selected and recommended according to each patient’s conditions based on the data elements (patient characteristics) that were chosen by specialists and system decision-making rules. The existing paper layout and content were mirrored to facilitate the required training, but the optimization was implemented for device sizes.

Evaluation results

A comparison of the system-based and paper-based documentation showed that the documentation of 150 cases (100%) was completed in 150 collected records in the system-based method, while 118 cases (78.66%) of all cases were completely documented in the paper-based method. One or more items of missing data were identified in 32 records. The missing data included NIHSS (5), therapeutic trend (16), and data related to date, time, and documentary identity (11). The system accuracy (in the test data) was 98.30%. The system sensitivity and specificity were also 98.29% and 100%, respectively [Table 2]. Evaluation results indicated that the severity of acute stroke and therapeutic trend of 1.69% of the patients were mistakenly determined, and in the traditional method, it was statistically significant (P < 0.05).

The Durbin–Watson statistic was 1.85 when the dependent variable was adherence to the guideline. This shows that there was no autocorrelation. Before the beginning of the data collection period, the mean of the adherence to the guideline was 65%. Before the intervention, there was no significant change in the trend of the adherence to the guideline (P < 0.27). After the system intervention, the mean of the adherence to the guideline significantly increased from 65% to 99.5% (P < 0.0001). Since the decision-making rules of the system are based on guidelines, and the system based on these guidelines guides the specialist physician at each stage, as a result when a specialist physician uses this system, the adherence to the guidelines increases. It is obvious when using a traditional method (paper-based). Guidelines may not be followed at some stage.

According to the results of the system evaluation using MARS questioners, the average score of system applicability was 4.6; system performance was 4.75; system esthetics was 4.66; and system information quality was 4.42. In general, the mean quality score of the system was 4.60 and the subjective quality score was 3.75, indicating the excellent and acceptable system quality [Table 3].

Discussion

The present study consisted of an intelligent information system for the severity assessment and determination of a therapeutic trend for the acute stroke. In this study, 53 data elements were identified for the severity assessment and determination of a therapeutic trend for the acute stroke. Identifying and determining data elements are the main tasks of data collection to achieve the appropriate functionality of intelligent systems.[15] A review showed that there was no standard and uniform format for data collection in this field.[16] Therefore, it is essential to determine the data elements for mobile-based intelligent systems for severity assessment and determination of a therapeutic trend for acute stroke patients at early stages.

Various methods have been suggested for assessing acute stroke severity in recent studies. Among them, the NIHSS is the most common method. A system was developed to diagnose the stroke severity based on the NIHSS in a study by Rajan et al.[17] This study was consistent with the present study for assessing the stroke severity, but it does not analyze the severity and does not provide any treatment, while the present research included all of these advantages. The mentioned study did not report the accuracy. Therefore, it can be claimed that the present system improved the compliance with acute stroke management tools and guidelines with a high accuracy .

A smartphone platform was created for the triage of patients with stroke in a study by Nogueira et al.[18] The above-mentioned system was compatible with the designed system of the present study in terms of their platforms, but they vary in terms of their application. The smartphone is a complex tool connecting people to a world of information. In recent years, the use of smartphones has been significantly growing. Smartphone features such as interactive screens, fast and easy access, data transfer and tracking, and pervasive influence have led to the more common use than other equipment for access to the internet and use in health applications.[19]

Results indicated that the documentation quality increased from 78.66% to 100%. A review of studies shows that completeness was the most commonly assessed dimension of data quality as an area of focus in 64% of papers.[20] Potential benefits of an intelligent system in the healthcare documentation include the improved quality of documentation, increased communication between users, reduced paperwork, and cost-saving. Electronic records allow for the real-time access that leads to faster data searches and increased physician efficiency.[21]

Our results show that the system improved adherence to the guideline for the severity assessment and determination of a therapeutic trend for acute stroke patients. This improvement may have occurred because the system reduced guideline complexity by simplifying calculation and interpreted the risk scores.[22] This result was similar to those of some other studies investigating the effect of intelligent systems on adherence to the guidelines. In one study, computer-assisted, nurse-driven, and guideline-based decision support system (DSS) was developed as an intervention. In this trial, the adherence to the guidelines was 96% in the intervention group compared to 70% in the control group (P < 0.001).[23] Contrary to these results, in another study, a DSS was developed to improve guideline adherence in the treatment of atrial fibrillation. There was no significant difference between the intervention groups and the control groups. Lack of clinical DSS use (5%), alert fatigue, and need to click to access information was stated as reasons for lack of effect.[24]

Most studies, which focused on information systems in the field of stroke, were developed with aims such as detecting the risk of stroke, helping to rehabilitate patients, and triage of stroke patients and compared the performance of different clinical methods. For instance, a study by Mehdipour et al. proposed a model to prediction of cerebrovascular accident. The comparison of results with neurologists’ opinions indicated an acceptable result.[25] Other studies have been also conducted using nonlinear learning models such as the neural network, Nave Bayesian, and support vector machine.[26,27]

According to the compared results of the present study with studies above, the proposed system had higher accuracy than other systems. All of these studies predicted the stroke risk using machine learning (ML) algorithms, while the proposed system used rule-based reasoning methods instead of ML algorithms, due to the nature of acute stroke diagnosis and treatment process. A rule-based reasoning approach is a well-known method that is applicable to design evidence-based expert systems. Results of some studies indicate that a rule-based reasoning approach has high accuracy (nearly 100%) compared with other ML methods.[28] The current study indicated that the domain knowledge should be available to developers in the evidence-based diagnosis and guideline-based medicine, because rule-based reasoning methods are more efficient in designing and they are easy-to-understanding for stakeholders. Some ML algorithms such as the artificial neural network behave as a black box, and the inference process is unclear for being interpretation by experts. However, a rule-based reasoning algorithm performs like a white box. It is important to analyze the problem-solving path in many health challenges. Owing to the uncertainty of medicine, rule-based reasoning methods help to analyze relations of rules and outcomes.[29]

Limitation

Coordinating the time of meeting with specialized physicians to extract and approve decision-making rules due to their busy schedule was one of the most important limitations of the present study, which was best managed with careful planning by the research team.

Conclusion

A system based on guidelines and clinicians’ opinions was designed and evaluated in the present study. It was found that designed system effectively assessed the severity of acute strokes and determined its therapeutic trends. This system reduced medical errors and improved the quality of documentation and adherence to the guideline. Therefore, due to the high prevalence of acute stroke in the world, the use of intelligent systems in the field of acute stroke diseases empowers physicians to assess and treat acute stroke and improve the quality of emergency services.

How to cite this article: Torab-Miandoab A, Samad-Soltani T, Shams-Vahdati S, Rezaei-Hachesu P. An intelligent system for improving adherence to guidelines on acute stroke. Turk J Emerg Med 2020;20:118-34.

This article does not contain any studies with human participants or animals performed by any of the authors. The Ethics Committee of Tabriz University of Medical Sciences has confirmed this research.

S SV and P RH conceived the original idea. A TM, T SS, and P RH designed the study. A TM and S SV collected data. A TM and T SS analyzed and interpreted the data. A TM, T SS, and P RH prepared the first manuscript draft. All authors contributed significantly and critically to the final manuscript.

None declared.

None declared.

References

1. Birtane M, Taştekin N. Quality of life after stroke. Med J Trakya Univer 2010;27:63-8.
2. Brust JC. Current Diagnosis and Treatment Neurology. New York: McGraw Hill Professional; 2011.
3. Roudbary SA, Saadat F, Forghanparast K, Sohrabnejad R. Serum C-reactive protein level as a biomarker for differentiation of ischemic from hemorrhagic stroke. Acta Medica Iranica. 2011:149- 52.
4. MEMBERS WG, Roger VL, Go AS, Lloyd-Jones DM, Benjamin EJ, Berry JD, et al. Heart disease and stroke statistics—2012 update: a report from the American Heart Association. Circulation. 2012 Jan 3;125:e2.
5. Saposnik G, Johnston SC. Decision making in acute stroke care: Learning from neuroeconomics, neuromarketing, and poker players. Stroke 2014;45:2144-50.
6. Rothschild JM, Hurley AC, Landrigan CP, Cronin JW, Martell-Waldrop K, Foskett C, et al. Recovery from medical errors: The critical care nursing safety net. Jt Comm J Qual Patient Saf 2006;32:63-72.
7. Handel D, Epstein S, Khare R, Abernethy D, Klauer K, Pilgrim R, et al. Interventions to improve the timeliness of emergency care. Acad Emerg Med 2011;18:1295-302.
8. Mankad KB. An intelligent process development using fusion of genetic algorithm with fuzzy logic. In: Artificial Intelligence: Concepts, Methodologies, Tools, and Applications. IGI Global; 2017. p. 245-81.
9. Abu-Nasser BS, Abu Naser SS. Rule-based system for watermelon diseases and treatment. Int J Acad Inf Syst Res 2018;2:1-7.
10. Lehmann CU. Medical information systems in pediatrics. Pediatrics 2003;111:679.
11. Arayici Y, Ahmed V, Aouad GF. A requirements engineering framework for integrated systems development for the construction industry. J Inf Technol Constr 2006;11:35-55.
12. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth. 2015;3:e27.
13. Nistal-Nuño B. Segmented regression analysis of interrupted time series data to assess outcomes of a South American road traffic alcohol policy change. Public Health 2017;150:51-9.
14. Kwah LK, Diong J. National Institutes of Health Stroke Scale (NIHSS). J Physiother 2014;60:61.
15. KalankeshLR,DastgiriS,RafeeyM,RasouliN,VahediL.Minimumdata set for cystic fibrosis registry: A case study in iran. Acta Inform Med 2015;23:18-21.
16. Martins SC, Martins MC, Carbonera LA, Souza AC, Portal M, Martin K, et al. Abstract TP312: Assessment of the data collection strategies for stroke patient-centered outcomes: Implementation of the International Consortium for Health Outcomes in Brazil. Stroke 2018;49 Suppl 1:ATP312.
17. Rajan V, Bhattacharya S, Shetty R, Sitaram A, Vivek G. Clinical Decision Support for Stroke Using Multi–view Learning Based Models for NIHSS Scores. InPacific-Asia Conference on Knowledge Discovery and Data Mining Springer, Cham. 2016. p. 190-9.
18. Nogueira RG, Silva GS, Lima FO, Yeh YC, Fleming C, Branco D, et al. The FAST-ED App: A Smartphone Platform for the Field Triage of Patients With Stroke. Stroke 2017;48:1278-84.
19. Fukuoka Y, Kamitani E, Dracup K, Jong SS. New insights into compliance with a mobile phone diary and pedometer use in sedentary women. J Phys Act Health 2011;8:398-403.
20. Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013;46:830-6.
21. Tsai J, Bond G. A comparison of electronic records to paper records in mental health centers. Int J Qual Health Care 2008;20:136-43.
22. Goud R, van Engen-Verheul M, de Keizer NF, Bal R, Hasman A, Hellemans IM, et al. The effect of computerized decision support on barriers to guideline implementation: A qualitative study in outpatient cardiac rehabilitation. Int J Med Inform 2010;79:430-7.
23. Hendriks JL, Nieuwlaat R, Vrijhoef HJ, de Wit R, Crijns HJ, Tieleman RG. Improving guideline adherence in the treatment of atrial fibrillation by implementing an integrated chronic care program. Neth Heart J 2010;18:471-7.
24. Arts DL, Abu-Hanna A, Medlock SK, van Weert HC. Effectiveness and usage of a decision support system to improve stroke prevention in general practice: A cluster randomized controlled trial. PLoS One 2017;12:e0170974.
25. Mehdipour Y, Ebrahimi S, Karimi A, Alipour J, Khammarnia M, Siasar F. Presentation a model for prediction of cerebrovascular accident using data mining algorithm. Sadra Med Sci J 2016;4:255-66.
26. Letham B, Rudin C, McCormick TH, Madigan D. An interpretable model for stroke prediction using rules and Bayesian analysis. InProceedings of 2014 KDD Workshop on Data Science for Social Good 2014.
27. Pardamean B, Christian R, Abbas BS. Expert-system based medical stroke prevention. J Comput Sci 2013;9:1099-105.
28. Nabaei A, Hamian M, Parsaei MR, Safdari R, Samad-Soltani T, Zarrabi H, et al. Topologies and performance of intelligent algorithms: a comprehensive review. Artificial Intelligence Review. 2018;49:79-103.
29. Sumarlinda S, Rahmat A, Long ZA. Clinical decision support system in computational methods: A review study. Proc ICOHETECH 2019;1:242-5.

This study as an MSc dissertation with Thesis number 59584/323 was done in School of Health Management and Medical Informatics of Tabriz University of Medical Sciences. Authors express their gratitude to emergency specialists at Imam Reza Hospital for their assistance and cooperation.