Article Text
PDF
Abstract
Introduction Artificial intelligence (AI) development has led to improvements in many areas of medicine. Canada has workforce pressures in delivering cataract care. A potential solution is using AI technology that can automate care delivery, increase effectiveness and decrease burdens placed on patients and the healthcare system. This study assesses the use of ‘Dora’, an example of an AI assistant that is able to deliver a regulated autonomous, voice-based, natural-language consultation with patients over the telephone. Dora is used in routine practice in the UK, but this study seeks to assess the safety, usability, acceptability and cost-effectiveness of using the technology in Canada.
Methods and analysis This is a two-phase prospective single-centred trial. An expected 250 patients will be recruited for each phase of the study. For Phase I of the study, Dora will phone patients at postoperative week 1 and for Phase II of the study, Dora will phone patients within 24hours of their cataract surgery and again at postoperative week 1. We will evaluate the agreement between Dora and a supervising clinician regarding the need for further review based on the patients’ symptoms. A random sample of patients will undergo the System Usability Scale followed by an extended semi-structured interview. The primary outcome of agreement between Dora and the supervisor will be assessed using the kappa statistic. Qualitative data from the interviews will further gauge patient opinions about Dora’s usability, appropriateness and level of satisfaction.
Ethics and dissemination Research Ethics Board William Osler Health System (ID: 22–0044) has approved this study and will be conducted by guidelines of Declaration of Helsinki. Master-linking sheet will contain the patient chart identification (ID), full name, date of birth and study ID. Results will be shared through peer-reviewed journals and presentations at conferences.
- Telemedicine
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Cataracts are a major cause of blindness globally, and artificial intelligence (AI), notably Dora, has shown promise in the UK for post-surgery management, addressing the need for innovative healthcare approaches.
WHAT THIS STUDY ADDS
This study evaluates the use of Dora in Toronto, Canada, for post-cataract surgery care, assessing its safety, acceptability, usability, cost-effectiveness and environmental impact.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The findings could lead to wider adoption of AI in healthcare, influencing research, practice and policy by demonstrating the benefits of AI in postoperative care and patient management.
Introduction
Cataract is defined as the degradation of the optical quality of the crystalline lens that affects vision and is the current leading cause of blindness worldwide.1 Age is a major factor in the development of cataracts, which can affect one or both eyes.2 According to the 2009 Cost of Vision Loss Report, cataracts were one of the top causes of blindness in Canada, affecting almost 3.5 million individuals.3 However, since 2006, the number of surgeries performed yearly has declined, and wait times have soared in Ontario, Canada.4 5 The reason for this declining trend after 2006 is likely multifactorial, including increasing cataract prevalence, restricted operational resources and a 5.9% decline in the number of cataract surgeons per 100 000 total population.5 There is a pressing need for strategies to support the workforce crisis in delivering high-volume, low complexity pathways such as cataract surgery.
The development of artificial intelligence (AI) has led to improvements in many areas of medicine.6 In ophthalmology, AI has shown promising results in the detection and screening of retinopathy of prematurity, age-related macular degeneration, glaucoma and diabetic retinopathy.7–11 Similar to most surgeries, cataract surgery involves postoperative monitoring for complications and to evaluate success. This visit has traditionally been carried out as an in-person visit. AI is set to revolutionise post-cataract surgery management by enhancing automation, increasing effectiveness and decreasing burdens placed on patients and the healthcare system.7 Ultimately, using AI-enabled automation could enhance patient management during and post-cataract surgery.
Dora is a regulated autonomous, voice-based, natural-language clinical assistant designed in the UK. It can have a consultation with patients over the telephone in a similar way to a human clinician by incorporating speech transcription, natural language understanding and a machine learning conversation model to allow contextual dialogues, speech production and natural conversation.12 The technology does not require the installation of an application, the provision of a device, or any training. This is relevant since elderly people and people from low-income families are more likely to be digitally excluded.12
Dora is used in the cataract pathway in the UK, and in a recent study, Khavandi et al found that the majority of patients regarded the AI-telephone follow-up after cataract surgery as very simple, easy to use and they appreciated the convenience.13 The patients also noted that an automated telephone follow-up might greatly lower the number of clinical appointments required to provide postoperative care because it would be less time-consuming than a clinician and would also free up clinicians’ time for other clinical tasks.12
This study is designed to further evaluate the impact of automating cataract follow-up using Dora, in a different Canadian healthcare setting. Technical readiness level of five meaning that technological components are currently being integrated for testing in relevant environments.12 14 The Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence reporting guideline was followed in creation of this protocol.15
The objectives of this study are to describe:
Clinical safety of using Dora in the cataract pathway using Kappa statistic of the interobserver decision reliability and retrospective review of clinical notes to establish whether the patient attended the clinic with further concerns after their Dora call.
Patient acceptability of Dora using Net Promoter Score (NPS) to determine on a scale of 1–10 the likelihood of them recommending Dora to a friend or colleague.
Usability and accessibility of the technology for patients using the scores from the System Usability Questionnaire (SUS).
Cost-effectiveness of using Dora compared with standard practice by comparing the cost of the Dora system to the clinic specific costs.
Impact on sustainability through use of the automated telephone follow-up by calculating travel distance and mode of transportation for each patient.
Methods and analysis
Study design
This study is a two-phase prospective single-centred trial. For Phase I of the study, Dora will phone patients at postoperative week 1 prior to their scheduled in-person visit and for Phase II of the study, Dora will phone patients on the day of cataract surgery within 24 hours following discharge home. Patients will continue to attend the standard week 1 in-person assessment during this study. On the day of their cataract surgery, patients will receive a printed sheet with contact details for emergency services, available both during and outside work hours, for immediate assistance. In Phase I, an ophthalmic technician will phone patients 24 hours after surgery to provide instructions on signs to watch for and how to access emergency care before the week 1 postoperative assessment. In Phase II, Dora will relay this information instead of the ophthalmic technician. In Phase I, patients who do not answer the Dora call will continue to attend their in-person postoperative week 1 visits. For Phase II, if patients miss their initial Dora call, an ophthalmic technician will contact them within 24 hours after their cataract surgery. If patients miss their in-person appointments, they will be contacted to reschedule. The key elements of the conversation with Dora are described below:
Greeting and introduction.
Confirmation of identity of patient.
Validated cataract follow-up questions.
Opportunity for patient to ask Dora questions
Decision regarding next steps of care.
Questions about acceptability (NPS).
Closure of call.
Every Dora call audio and transcript will be reviewed by an expert clinician. They will make an assessment on the significance of each individual symptom and on the overall call decision. The clinician will be blinded to Dora’s symptom assessments. The outcomes from a Dora call are defined as shown in table 1.
Outcomes from Dora call
Dora calls made on the day of surgery will evaluate if the patient has any acute postoperative concerns like pain and will prompt a postoperative day 1 visit if necessary. A clinician will listen to the Dora call after it is completed and they can either call the patient back if further information is required, arrange an additional in-person visit, or confirm that the patient can move onto the next stage of their pathway.
All patients will be seen at the planned postoperative week 1 (POW1) visit the day after their call with Dora at week 1. Details on clinical and examination findings will be recorded at this visit.
Patients will fill out a standardised paper questionnaire at their week 1 in-person visit to assess the system’s usefulness and acceptance, cost-effectiveness and sustainability. Within 3 weeks of the Dora call, another researcher-led call will be made to further assess the system’s usefulness and acceptance and produce thematic analysis regarding usability, appropriateness and satisfaction of Dora. The follow-up interview questions are consistent with current validated questionnaires of SUS and Theoretical Framework of Acceptability (TFA).16 17 This interview will also assess whether there have been any changes to the patient’s medical condition after the Dora call was completed. Patients will be grouped based on their ethnicity as self-reported on their medical chart and from each group a randomly selected population will be interviewed to better understand their experience with Dora. One-to-one interviews will be conducted using a semi-structured interview guide and an inductive thematic analysis approach will be undertaken to analyse the data. Individuals who do not have ethnicity reported on their medical chart will be pooled into one group and a random sample will be selected from that group as well. Study flow is presented in figure 1.
Study flow diagram. F2F, face-to-face, OT, ophthalmic technician, POW1, postoperative week 1.
Sample selection
Patients undergoing routine cataract surgery at Uptown Eye Specialists Surgical Centre, Vaughan, Ontario, Canada will be asked to participate in the study. Uptown Eye Specialists is a surgical centre which houses ophthalmologists that are affiliated with various greater Toronto area hospitals. The study will include 250 patients who are undergoing routine unilateral and/or bilateral cataract surgery either for their first, second eye or both. Exclusion criteria include patients identified as having intraocular conditions leading to complicated cataract surgery such as inflammatory conditions and glaucoma that require more than the standard postoperative treatment or follow-up,18 or patients requiring intraocular lens exchange or readjustment surgery.
Intervention
This study will use Dora (Ufonia Limited), a UK Conformity Assessed (UKCA)-marked clinical conversational assistant that uses speech recognition and natural language processing to have natural, voice conversations with patients over the phone.
Dora will call patients at two time points after their cataract surgery. For Phase I of the study, Dora will phone patients at POW1 and for Phase II of the study, Dora will phone patients on the day of cataract surgery (within 24 hours following discharge home).
Proportion and baseline characteristics of patients who agreed to participate and if provided, the reasons for declining to participate will be recorded. During the in-person week 1 visit which is after the Dora call, all participants will be asked to fill out a standardised paper questionnaire to assess satisfaction, usefulness, cost-effectiveness and sustainability of the system with the SUS consisting of 10 statements as well as additional demographic questions related to patients’ spoken language and burdens of travel.16 A random selection of patients will undergo additional semi-structured interviews to further assess the system’s appropriateness and satisfaction. The semi-structured free form interview with the participant will take between 15 and 30 min. The interview was developed based on the TFA, which was created to provide a framework for assessing the multiple facets of acceptability of health interventions. The TFA has seven components: (1) affective attitude, (2) burden, (3) ethicality, (4) intervention coherence, (5) opportunity costs, (6) perceived effectiveness, and (7) self-efficacy.17 An outline structure of the interview is presented in online supplemental appendix A.
Supplemental material
Sample size calculation
The study will include 250 patients who are undergoing uncomplicated unilateral and/or bilateral cataract surgery either for their first, second eye or both. This sample size was calculated drawing on previous studies on Dora as well as consultation with biostatistician from the institution of study.14 Estimated sample size of 250 to provide statistically significant outcome took into consideration the expected Kappa of 0.80 that shows a strong level of agreement between Dora’s decision and clinician’s grading.19 With the precision set at 0.1, allowing for a range in agreement from 0.70 (denoting a moderate level of agreement) to 0.90 (indicating a strong level of agreement). The assumption that 25% of patients would require additional in-person follow-ups was based on a review of 200 patients who had previously undergone cataract surgery at the Uptown Eye Specialists Surgical Centre. Additionally, we evaluate Dora’s specificity, which underscores Dora’s ability to accurately approve patients compared with the supervisor’s judgement and Dora’s sensitivity, which emphasizes Dora’s accuracy in identifying patients who did not meet the criteria as per the supervisor’s decision.
What outcomes will be measured, when and how and data analysis plan
Analysis of study outcomes has not begun yet. We will be using mean and SD for parametric data and median and IQR for non-parametric data. The details of the outcomes and planned data analysis plan can be found in table 2.
Outcome evaluation framework for Dora
Patient and public involvement
Patients were included in the design of this study and will be included in the analysis.
Ethics and dissemination
This study has been approved by the Research Ethics Board William Osler Health System (ID: 22–0044) and will be conducted in line with the guiding principles detailed in the Declaration of Helsinki. Informed consent will be obtained by study personnel at the time of booking of surgery. Master-linking sheet will contain a patient chart ID, full name, date of birth and study ID. This will be kept separate on a computer which is password protected and locked in an office. Chart ID and full name will be listed in the master-linking sheet only. Anonymised date of birth and gender will be collected for data analysis. Patients undergoing cataract surgery will be given information about the study and informed consent will be obtained. Surgical counsellors who are booking the patient for cataract surgery will review patient eligibility, give information about the study and obtain informed consent if eligibility criteria are met. We intend to share the results of both phases of the study through peer-reviewed journals, conference presentations and internal meetings. The research team will retain study records, patient files and other source data at least 7 years after the completion of the study.
The findings of the project will be summarised for a general audience, in accessible formats including written, audio and video formats. This will include local and national patient representative groups as well as information shared to patients on the cataract pathway.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by William Osler Health System REB# 22-0044. Participants gave informed consent to participate in the study before taking part.
Supplementary material
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors All of the authors have met the following criteria: Substantial contributions to the conception or design of the work; or the acquisition, analysis or interpretation of data for the work; drafting the work or revising it critically for important intellectual content; final approval of the version to be published; agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests SS: speaking honorarium for Allergan, Bayer, Novartis and industry ad board member for Novartis, Consultant Ripple Therapeutics. HHC: ad board member for Bayer, Novartis and Johnson & Johnson. Public and private support: nothing to disclose. AHi, SK, EL and NdP are employees of Ufonia, Limited.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.