A better approach is to clarify the health outcome being sought and determine whether existing data are available that can be rigorously and objectively evaluated, independently of or in comparison with data from RCTs, or whether new studies (RCT or otherwise) are needed.

every time they push out of their comfort zone to learn something new and difficult, the neurons in their brain can form new, stronger connections, and over time they can get smarter.

link between limited English language proficiency, which could explain some of the racial health disparity for Hispanic patients.

This method of analyzing physician work has far-reaching implications for payment reform. While it may be good or bad that physicians are spending more time documenting care and communicating with other staff members than they are in face-to-face visits with patients, that fact highlights the misalignment of a payment policy that reimburses only office visits, lab work, and procedures while overlooking much of desktop medicine work.

We found that the average EHR use time during an exam was 5.9 ± 2.3 min, and after the exam was 4.8 ± 4.7 min, for a total of 10.8 ± 5.0 min per appointment. This translates to a total of about 3.7 h EHR use time for a full-day clinic.

Patient wait time is a result of pressure on provider time as well as clinic inefficiency; wait time has been shown to affect patient satisfaction as well as create barriers to health care.21,22. Mathematics, specifically queueing theory, explains waiting by the mismatch of arrival times and service times (time with a physician).23 This mismatch can be increased by ad-hoc scheduling protocols that artificially increase patient wait time.24,25 Addressing this mismatch using smarter scheduling strategies has potential for improving patient wait time.

It was therefore impossible to distinguish between the time spent when a clinician was active in the EHR and the time spent while having the EHR open but engaging in other activities (eg, performing an examination, talking to a patient or team member), so we applied a cutoff of 90 seconds when no activity was captured by event log records as supported by the pause time observed in our direct observation period.

Physicians spent nearly one-half of their total EHR time per day (157 minutes, 44.2%) doing clerical tasks and an additional 84 minutes per day (23.7%) managing their inbox (Table 3).

average total EHR time per weekday for a 1.0 clinical FTE was 355 minutes (5.9 hours), consisting of 269 minutes (4.5 hours) during clinic hours and 86 minutes (1.4 hours) after clinic hours

he effectiveness and efficiency or brevity of available resources, and resource accessibility. “There’s a million things you could look up on patients. You have to stratify things.”65 Higher urgency encouraged seeking answers, while time was typically viewed as insufficient to tackle questions of greater complexity. “Patients are so much more complex than they were 20 years ago. … There’s actually less time and more pressure.”41

The program should be able to offer different kinds of information (diagnostic vs. educational) based on the user’s specific need at that moment.33

The first step in mitigating this pain point could be in understanding where users begin their searches. If searches begin in the EHR, resource integration and access should be made easier, more visible, and more efficient [7, 17]

Integration is a pain point that includes integrating information resources into clinicians' workflow. The integration pain point is closely linked to time because improved integration is a good way to improve efficiency.

Issues involving too many platforms and lack of standardization across those platforms encompass the next pain point: resource platforms. Each publisher platform can have different search functions, rules of access, and accessibility [26]. Users may not search the full breadth of content that they have access to because they have to search across multiple platforms.

time, which includes time to get access, search, find, and receive help. Clinicians see many patients a day and spend a lot of time on documentation and reporting. Research has shown that clinicians lack the time needed to answer clinical questions and read up on useful and of-interest topics [7, 13]. Potential solutions include integrating information resources into the clinician workflow or electronic health record (EHR) [14], implementing library consult services [15], and providing continued yet improved, easy, and timely access to librarians and informationists.

A 1985 study found that only 30% of physicians' information needs were met during a typical day [5]. In 2014, a systematic review analyzing 11 studies reported that clinicians generated about 1 question for every 2 patients and pursued answers to about 50% of them, but more than 20% of these questions went unanswered [6].

In one of the largest studies ever conducted on the value of libraries and information resources, 59% of physicians, residents, and nurses searching for information on patient care in electronic journals, PubMed/MEDLINE, and point-of-care tools, among many other resources, completely found what they were looking for, and over 95% of the more than 16,000 respondents found the information provided in the resources to be relevant, accurate, and current [4]. Furthermore, 33% said their choice of drugs was impacted by the information, and 19% said the information reduced unnecessary tests and procedures.

To the extent that the increase in patient messages is durable, clinicians will need to allocate more of their time to answering those messages, even after the pandemic has subsided. This is especially important given recent evidence identifying In-Basket messages in particular as a predictor of physician burnout, as well as other studies finding a significant increase in patient messages received by clinicians during the pandemic.

For Clinical Review, it may be that telemedicine encounters led to an increased reliance on previous documentation and viewing of test results in the EHR, or the increase in messages required clinicians to go back and review their own documentation before responding to patients.

The fact that clinicians spent more time working in the EHR during the COVID-19 pandemic is unsurprising but still concerning. An obvious potential driver of this finding is that many clinicians delivered care virtually via telemedicine, including both phone and video visits. The rapid proliferation of virtual visits may have had implications not only for how clinicians deliver care during the encounter but also how patients expect to interact with their clinician outside of the boundaries of the scheduled appointment.

Our multivariate fixed effects regressions suggest that each patient message increased In-Basket time per day by 2.32 min (P < .001, 95% CI: 2.16–2.48). Each Results message also significantly increased In-Basket time per day by 0.24 min (P < .001, 95% CI: 0.20–0.28). No other message type was associated with a statistically significant increase in In-Basket time per day (Table 1).

clinicians receiving 4% more messages the week of July 5, 2020, compared with the 11-week prepandemic average (Figure 4). The greatest increase was in messages from patients, with clinicians receiving 157% of their prepandemic average per day; that increased level of messages remained consistent through the end of the year. Team and prescription messages also remained at higher levels (relative to baseline) during the postpandemic period after July 5, 2020, whereas results messages remained below the prepandemic average after July 5, 2020. Other message types saw a similar pattern to the results messages, and raw message counts were also similar (Supplementary Appendix Exhibit SA2).

The method proposed here aims to correct the type I error (erroneously rejecting the null hypothesis) level, most likely at the cost of vastly increasing the number of type II errors (erroneously rejecting the alternative hypothesis).

Our proposed approach instead derives an empirical null distribution from the actual effect estimates for the negative controls. These negative control estimates give us an indication of what can be expected when the null hypothesis is true, and we use them to estimate an empirical null distribution. We fitted a Gaussian probability distribution to the estimates, taking into account the sampling error of each estimate. We have found that a Gaussian distribution provides a good approximation, and more complex models, such as mixtures of Gaussians and non-parametric density estimation, did not improve results.

Step 1 requires knowledge to construct a question using the PICO mnemonic, Step 2 requires the acquisition and application of literature searching skills across a variety of databases, Step 3 requires a certain level of expertise in epidemiology and biostatistics, and Step 4 requires an ability to synthesise and communicate the results to relevant parties (i.e. health professionals, patients). Step 5 requires the health professional to evaluate the EBP process and assess its impact within the clinical context in which it was implemented.[5]

Sicily Statement puts forward a five-step model: (I) asking a clinical question; (II) collecting the most relevant evidence; (III) critically appraising the evidence; (IV) integrating the evidence with one’s clinical expertise, patient preferences and values to make a practice decision; and (V) evaluating the change or outcome [4].

“The curiosity we feel when we see something that is surprising or puzzling or ambiguous, that doesn’t agree exactly with our previous knowledge or presumed knowledge, is not the same as the curiosity we feel as the love of knowledge — what drives research in science, for example. The first one is associated with a state of mind that is aversive. It’s an unpleasant feeling, which we try to get rid of.”

ASCVD events occur at LDL-C, non-HDL-C, and triglyceride levels that are even modestly elevated, further substantiating the importance of identifying and treating these patients early, who may be at a higher risk of ASCVD events than previously acknowledged.

Overall, a wide body of evidence suggests that the presence of gestational complications, premature menopause, and ovarian failure is associated with a higher lifetime risk of ASCVD. These findings highlight the importance of taking a thorough obstetrical history and obtaining information regarding age of menopause to identify those women who may need early and aggressive lipid lowering therapy in addition to lifestyle modification.

Several conditions specific to women have been identified as risk-enhancing factors in the 2018 guidelines. These include premature menopause, premature ovarian failure, pre-eclampsia, gestational hypertension, gestational diabetes, pre-term delivery, and delivery of small for gestational age infants.

Taken together, patients with South Asian ancestry or a family history of premature ASCVD are at an increased risk for future ASCVD events and clinicians should take these factors into consideration during risk stratification.

Embrace candor Working through so much change and dealing with unexpected setbacks means we need to be constantly and honestly communicating with one another to co-create the right new norms and habits.

Believe that everything is going to work out just fine, while accepting that getting there might not be easy. Research consistently shows that having positive expectations — or as pioneering social psychologist Albert Bandura called it, a strong sense of self-efficacy — is essential for staying motivated in the face of obstacles and setbacks.

Begin slowly building your resilience bank accountMaddaus’ idea of a resilience bank account is gradually building into your life regular practices that promote resilience and provide a fallback when life gets tough. Though it would obviously be nice to have a fat account already, he says it’s never too late to start. The areas he specifically advocates focusing on are sleep, nutrition, exercise, meditation, self-compassion, gratitude, connection, and saying no.

Expect less from yourselfMost of us have heard for most of our lives to expect more from ourselves in some way or another. Now we must give ourselves permission to do the opposite. “We have to expect less of ourselves, and we have to replenish more,” Masten says. “I think we’re in a period of a lot of self discovery: Where do I get my energy? What kind of down time do I need? That’s all shifted right now, and it may take some reflection and self discovery to find out what rhythms of life do I need right now?”

Surge capacity is a collection of adaptive systems — mental and physical — that humans draw on for short-term survival in acutely stressful situations, such as natural disasters. But natural disasters occur over a short period, even if recovery is long. Pandemics are different — the disaster itself stretches out indefinitely.

email inbox itself has become a symbol of stress and overload. Combine that with a 2012 McKinsey report that found employees spend approximately 28% of their time in the office responding to, reading, or composing emails. The average person checks his or her email upwards of seventy times per day, and on the high end that number approaches 350 times! Companies need to pay attention.

Methods need to be fit for purposes, according to session moderator Nancy Dreyer, chief scientific officer and senior vice president at IQVIA. “Early in the pandemic, high-profile retraction of studies using real-world evidence fueled mistrust.”Dreyer was referencing moves last year by The Lancet and the New England Journal of Medicine, which retracted separate studies relying on the same international database that included electronic health records (EHR) from 169 hospitals. The studies hit speed records by making it from last patient visit to publication in under six months, she says, but had troubling issues with source data validation.

There is moderate supporting evidence of associations between message use and selected patient outcomes (eg, glucose levels in patients with diabetes), and some evidence for other outcomes (eg, diastolic and systolic blood pressure among patients with hypertension) [Goldzweig CL, Orshansky G, Paige NM, Towfigh AA, Haggstrom DA, Miake-Lye I, et al. Systematic Review: Secure Messaging Between Providers and Patients, and Patients' Access to Their Own Medical Record: Evidence on Health Outcomes, Satisfaction, Efficiency and Attitudes. Washington (DC): Department of Veterans Affairs (US); 2012.44]. We observed significant differences in email communication by individuals based on education, race, ethnicity, and insurance status, with patients with lower levels of education, black patients, those with Medicaid or other public payers, and uninsured patients having reduced odds for secure messaging use. Such differences in the use of a communication modality that might have positive impacts on health outcomes—which permits patients to communicate with clinic staff at their convenience and can increase satisfaction and improve understanding of their condition [Crotty BH, Tamrat Y, Mostaghimi A, Safran C, Landon BE. Patient-to-physician messaging: volume nearly tripled as more patients joined system, but per capita rate plateaued. Health Aff (Millwood) 2014 Oct;33(10):1817-1822 [FREE Full text] [CrossRef] [Medline]6,North F, Crane SJ, Stroebel RJ, Cha SS, Edell ES, Tulledge-Scheitel SM. Patient-generated secure messages and eVisits on a patient portal: are patients at risk? J Am Med Inform Assoc 2013;20(6):1143-1149 [FREE Full text] [CrossRef] [Medline]45]—may further exacerbate health disparities if not addressed.

Our findings demonstrate that most physicians have secure messaging capabilities, but patients were not taking advantage to communicate using that modality with this clinic staff.

Patient activation follows 4 stages: belief in the importance of engagement in the care processes; knowledge in what is needed to improve health; taking action to improve or maintain health; and finally, maintaining or persisting in those actions even when stressed [Hibbard J, Stockard J, Mahoney E, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004 Aug;39(4 Pt 1):1005-1026 [FREE Full text]31].

Our research found associations between selected message content and changes in patients’ glycemic levels and blood pressures. As we anticipated, patients who sent Information seeking messages, or who received Orientation to procedures or treatments messages from clinic staff, experienced greater decreases in A1C. Consistent with other research [Harris LT, Haneuse SJ, Martin DP, Ralston JD. Diabetes quality of care and outpatient utilization associated with electronic patient-provider messaging: a cross-sectional analysis. Diabetes Care 2009 Jul 14;32(7):1182-1187 [FREE Full text] [CrossRef] [Medline]12,Zhou YY, Kanter MH, Wang JJ, Garrido T. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health Aff (Millwood) 2010 Jul 20;29(7):1370-1375. [CrossRef] [Medline]14,Price-Haywood EG, Luo Q, Monlezun D. Dose effect of patient-care team communication via secure portal messaging on glucose and blood pressure control. J Am Med Inform Assoc 2018 Jun 01;25(6):702-708. [CrossRef] [Medline]16-Chung S, Panattoni L, Chi J, Palaniappan L. Can Secure Patient-Provider Messaging Improve Diabetes Care? Diabetes Care 2017 Oct;40(10):1342-1348. [CrossRef] [Medline]19], we found an overall association between secure messaging use and improved A1C.

Patients had to have at least one inpatient or two outpatient visits in 2016 with ICD-10-DM diagnosis codes for either diabetes (E11) or hypertension (I10), and one visit in 2018.

Based on Mishel’s Uncertainty in Illness Theory [Mishel MH. Uncertainty in illness. Image J Nurs Sch 1988;20(4):225-232. [CrossRef] [Medline]23] and Street et al’s [Street RLJ, Makoul G, Arora NK, Epstein RM. How does communication heal? Pathways linking clinician-patient communication to health outcomes. Patient Educ Couns 2009 Mar 23;74(3):295-301. [CrossRef] [Medline]11] framework, we anticipated that actions indicative of self-care, such as Task-oriented requests not associated with uncertainty (eg, routine appointment requests, prescription refills) and Self-reporting, would be associated with improved health outcomes. Similarly, actions from the patients that might be indicative of trust between patient and clinician, such as Information sharing and positive Social communication, would be associated with improved health outcomes. Clinicians’ Information sharing and Encouragement could be a mechanism to mitigate patients’ uncertainty and improve trust, so we hypothesized that receipt of this message content would also result in improved health outcomes.

These mixed findings, particularly the findings from Shimada et al [Shimada SL, Allison JJ, Rosen AK, Feng H, Houston TK. Sustained Use of Patient Portal Features and Improvements in Diabetes Physiological Measures. J Med Internet Res 2016 Jul 01;18(7):e179 [FREE Full text] [CrossRef] [Medline]17], suggest that we must move beyond counting messages and begin classifying and quantifying message content types in order to better understand how secure messages, and more specifically, secure messaging content, impact important patient health outcomes. This research study directly attempts to address this need.

Increasingly, evidence indicates that the EHR is imposing an intolerable burden on clinicians and may be degrading, rather than elevating, clinical care. A study of EHR use measurements across 2 vendor products found that ambulatory, nonteaching physicians (n = 573) spent more than 5 hours on the EHR for every 8 hours of scheduled clinical time.1 Clinicians trained in patient care are locked into hours of screen time to complete mandatory clerical and documentation tasks, often unrelated to the quality of the care.

(1) have evidence linking it to a particular outcome, (2) target an area with poor performance and/or wide variation, and (3) produce actionable results that are usable and relevant to the needs and interests of specific stakeholders including clinicians, patients, practice leaders, vendors, and policy makers.

Highlighting how much time is spent on documentation and other EHR tasks not directly related to clinical care could inform local redesign and future policy regarding the documentation, billing, or regulatory requirements.

not all health care systems study patient portal utilization systematically; thus, health care system support of different communication modalities is essential.

A total of 30 studies in our systematic review specifically analyzed provider use, with nearly half of those studies analyzing provider-initiated messages. This fact further highlights the notion of physician use predicting patient use of portal functions and the intersection of physician portal use with institutional support of portal utilization. Recognizing that organizational policies mandate physician portal use, and nonuse, future studies should examine metrics for provider use more distinctly from patient use. As mentioned previously, provider use was only analyzed separately from patient use in 6% of studies, generating an untapped lens through which to investigate the driving forces of patient portal utilization. Mafi et al [Mafi JN, Mejilla R, Feldman H, Ngo L, Delbanco T, Darer J, et al. Patients learning to read their doctors' notes: the importance of reminders. J Am Med Inform Assoc 2016 Sep;23(5):951-955 [FREE Full text] [CrossRef] [Medline]45] found that individualized physician reminders to patients alerting them of completed visit notes drove patient portal utilization and engagement.

Further, initiation of portal use has been found to be “lower for racial and ethnic minorities, persons of lower socioeconomic status, and those without neighborhood broadband internet access,” leading to a digital divide in portal utilization [Perzynski AT, Roach MJ, Shick S, Callahan B, Gunzler D, Cebul R, et al. Patient portals and broadband internet inequality. J Am Med Inform Assoc 2017 Sep 01;24(5):927-932 [FREE Full text] [CrossRef] [Medline]63].

Numerous studies have investigated the relationship between patient portal utilization and health outcomes, specifically indicating a link between increased portal use and increased rates of patient engagement [Price-Haywood EG, Luo Q. Primary Care Practice Reengineering and Associations With Patient Portal Use, Service Utilization, and Disease Control Among Patients With Hypertension and/or Diabetes. Ochsner J 2017;17(1):103-111 [FREE Full text] [Medline]6-Lyles CR, Sarkar U, Schillinger D, Ralston JD, Allen JY, Nguyen R, et al. Refilling medications through an online patient portal: consistent improvements in adherence across racial/ethnic groups. J Am Med Inform Assoc 2016 Apr;23(e1):e28-e33 [FREE Full text] [CrossRef] [Medline]9]. Notably, engaged individuals more actively participate in the management of their health care [Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient Portals: An Underused Resource for Improving Patient Engagement. Chest 2016 Jan;149(1):272-277. [CrossRef] [Medline]10] and report enhanced patient satisfaction [Dendere R, Slade C, Burton-Jones A, Sullivan C, Staib A, Janda M. Patient Portals Facilitating Engagement With Inpatient Electronic Medical Records: A Systematic Review. J Med Internet Res 2019 Apr 11;21(4):e12779 [FREE Full text] [CrossRef] [Medline]11], a finding increasingly critical in patients with chronic diseases [McAlearney AS, Sieck CJ, Hefner JL, Aldrich AM, Walker DM, Rizer MK, et al. High Touch and High Tech (HT2) Proposal: Transforming Patient Engagement Throughout the Continuum of Care by Engaging Patients with Portal Technology at the Bedside. JMIR Res Protoc 2016 Nov 29;5(4):e221 [FREE Full text] [CrossRef] [Medline]12]. Patient portal utilization has been linked to “significant decreases in office visits…, changes in medication regimen, and better adherence to treatment” [Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015 Feb 10;17(2):e44 [FREE Full text] [CrossRef] [Medline]13], along with improved chronic disease management and disease awareness [Sarkar U, Lyles CR, Parker MM, Allen J, Nguyen R, Moffet HH, et al. Use of the refill function through an online patient portal is associated with improved adherence to statins in an integrated health system. Med Care 2014 Mar;52(3):194-201 [FREE Full text] [CrossRef] [Medline]8,Lyles CR, Sarkar U, Schillinger D, Ralston JD, Allen JY, Nguyen R, et al. Refilling medications through an online patient portal: consistent improvements in adherence across racial/ethnic groups. J Am Med Inform Assoc 2016 Apr;23(e1):e28-e33 [FREE Full text] [CrossRef] [Medline]9]. Interestingly, even the content of patient messages was recently found to be associated with estimated readmission rates in patients with ischemic heart disease [Sulieman L, Yin Z, Malin BA. Why Patient Portal Messages Indicate Risk of Readmission for Patients with Ischemic Heart Disease. AMIA Annu Symp Proc 2019;2019:828-837 [FREE Full text] [Medline]14]. In these ways, patient portals have been cited as essential components of the solution to the cost and quality health care crisis in the United States [Irizarry T, DeVito DA, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res 2015 Jun;17(6):e148 [FREE Full text] [CrossRef] [Medline]2].

Previous work within the Department of Veterans Affairs found that health care professionals receive large quantities of EHR-based notifications, making it harder to discern important vs irrelevant information and increasing their risk of overlooking abnormal test results.3-6 Information overload is of emerging concern because new types of notifications and “FYI” (for your information) messages can be easily created in the EHR (vs in a paper-based system).

better understand the relative contribution of cognitive load to a variety of outcomes. These might include not only measures of well-being (for example, burnout), but other psychological consequences such as depression or anxiety, and additional targets, such as medical errors and related quality indicators.

Another strategy explored in the clinician well-being space that seems well-suited to decrease cognitive load is the use of team-based documentation and workflow strategies, particularly to target the inbox. Although messages related to clinical decisions are relevant to the practicing clinician, they are often accompanied by other inquiries best handled by other members of the health care team, such as medication refill requests, scheduling or billing inquiries, and nonclinical matters. Even if a resource exists for the physician to divert these tasks, the individual items summate to a considerable overall task load. Team-based care and next-generation EHR solutions are necessary to divert this load away from the clinician, who is better served to address primarily clinical issues.

This pandemic-related cognitive load may further be hastened and moderated by the compounded stressful effects of uncertainty4 and the impact of unrelenting increased work demands, resulting in a fatigued and beleaguered workforce.

Health care today seemingly unleashes innumerable sources of cognitive load on the physician across all four domains described in the authors’ study. Those categorized as extraneous, or related to the way information is organized when presented, are perhaps the most amenable to intervention. Examples of such extraneous load include, but are not limited to, electronic health record (EHR) inbox notifications, patient encounter documentation requirements, and an overwhelming array of data and communications streams.

investigators observed a dose-response relationship in which the odds of experiencing burnout decreased with decreasing scores on the task load measure. Though some specialties experienced considerable variation in levels of cognitive load, it was nonetheless observed to correlate with burnout independent of specialty.

Specific key challenges related to the theme were identified during the June 2018 HTAi GPF scoping meeting, and include: (i) quality and acceptability, (ii) governance and accountability, (iii) transferability, and (iv) informing decision making. The key challenges, as well as different initiatives in the field of RWD/RWE, were described in a background paper that served as an input to the 2019 HTAi GPF (Reference Oortwijn15). In this article, we provide a reflection of the discussions of the above-mentioned challenges and the opportunities for addressing these that occurred during the 2019 HTAi GPF. This article is not a consensus statement of the attendees. As such, it cannot be taken to represent the views of any of the individuals attending the meeting or of the organization for whom they work.

conducted their own research, only to be dismissed by the clinician. A 26-year-old female sent her clinician information about side effects, which resulted in a response stating that she was “not the doctor.” Participants praised the clinician’s response in the partnership example, which included a patient searching for information using the internet. A 71-year-old male called this response, “The ultimate goal,” and a 28-year-old thought it demonstrated “compassion.”

“It shows me what kind of doctor I have,” and a 21-year-old female expounded, “I would be very frustrated if I received a message like that…I would immediately call them and say, ‘you’re going to see me right now.’” Participants expected more thoughtful communication, summarized by a 44-year-old male who stated, “I think in the 21st century, email is a part of bedside manner now.”

According to participants, receiving a message about their health from a clinician was a significant correspondence, and the content of the reply can have several consequences, identified by the following sub-themes: (1) altering the patient-clinician relationship, (2) lack of PCC can produce uncertainty, and (3) comparisons to face-to-face interactions.

The majority of participants echoed this notion and believed that, if clinicians were more accepting of SM, it would become a time-saver for them as well. Moreover, effective communication using SM prevented participants from booking unnecessary appointments.

“I hate the phone. Because everybody’s talking over each other and it’s just very difficult to communicate that way.” Participants also lauded the benefits of SM to expedite their care. A 28-year-old female recalled the time in which she initially found out she had a tumor. After sending a SM, she remembered, “My doctor got back to me right away. She’s like, ‘come in. Let’s figure out what to do.’ It went quicker rather than me calling and waiting on the phone.” Similarly, a 64-year-old male appreciated the ability to communicate with a clinician on his own terms. He said: If I can do it at my particular time and move on and know that 24 hours later I will have an answer, I would prefer to do it that way…Because there’s nothing worse than having to sit there on the phone and wait.

CUMULATIVE DISCUSSION AND CONCLUSIONS

We built and evaluated five multi-label machine learning classifiers to identify communication types in secure messages between providers and staff. Machine learning classifiers included random forest, multinomial naïve bayes, support vector machine (SVM), bidirectional encoder representations for transformers (BERT) [214], and clinical BERT, a BERT model that was previously trained on a corpus of clinical notes.

CLASSIFICATION AND ANALYSIS OF ASYNCHRONOUS COMMUNICATION CONTENT BETWEEN CARE TEAM MEMBERS INVOLVED IN BREAST CANCER TREATMENT

Our taxonomy was adapted from the parent categories of the Taxonomy of Consumer Health Information Needs[29]

THE HIDDEN WORK OF CLINICAL COMMUNICATION IS PRIMARY WORK OF ONCOLOGY CARE TEAMS: AN ANALYSIS OF ASYNCHRONOUS MESSAGING

Discussion

Our study shows that administrative and non-physician clinical staff perform 76% of the total clinical messaging work and that clinical messaging is included in 57% and 42% of their EHR sessions respectively. These results suggest that asynchronous messaging work has evolved from hidden tasks that support care to become the primary work that is integral to delivering and coordinating care for team members across all roles

We combined secure message logs, EHR audit logs, and breast cancer provider appointment logs to conduct a social network analysis and investigate how asynchronous clinical communication contributes to the frequency and duration of EHR work across a care team

esearch to date focused on quantifying care team collaboration has relied on identifying shared patients, but does not incorporate the major roles of communication patterns.[40,128,144,158]

appointment data corresponding to an appointment after cancer diagnosis. Appointment data included a unique patient identifier, a unique provider identifier, and an appointment date. Patients who had at least one outpatient appointment in the six-months following their diagnosis were included in the study. We also extracted secure clinical messaging logs from the EHR to understand the scope of communication between providers after the patient’s diagnosis between January 1, 2011 and November 1, 2017.

o analyze relationships between providers, we created two types of networks: a network of all providers involved in the care of a patient in our cohort and a network of providers involved directly in breast cancer treatment and their immediate connections.

Similarly, the onset of professional exhaustion and burnout from administrative work has led healthcare institutions to consider opportunities to reduce unnecessary and inefficient demands of the complex clinical environment. Despite the recognized need to identify and resolve communication inefficiencies, there remains a need for a scalable and data-driven framework to investigate the electronic work of clinical communication.

The time-motion study design was originally designed as a business efficiency technique. [114] As a result, these studies primarily focus on the efficiency of communication. [115]

Many early studies to quantitatively investigate clinical communication applied time-motion analyses to monitor and track communication and information transfer in localized environments. [13,89,110-113]

Our taxonomy employs concepts from both the UIT7 uncertainty antecedents constructs and the Street, Makoul, Arora, and Epstein31 clinical communication functions indirectly and directly associated with patient outcomes. In this way, we were able to identify content likely associated with patients' uncertainty (eg, Information seeking taxa) or self-management (eg, Task-oriented requests) as well as clinical responses linked to communication functions that are part of patient-centered care.

The most common type of content was Information sharing (59.29). Orientation to processes & procedures accounted for 55.34% of staff-generated Information sharing content.

Information seeking content was the most common and appeared in almost 30% of messages. Information seeking/Medical guidance was included in almost three-quarters (71.79%) of all initiating message

Consistent with the UIT, we expected patients would use secure messaging to communicate with clinic staff between visits to address uncertainty in their health status and manage their care when not uncertain about their health status.

The adjective “pure” is justified by the algorithms’ focus on prediction, to the neglect of estimation and attribution.

Column 3 shows standard two-sided p-values for the 11 variables, 6 of which are significantly nonzero, 5 of them strongly so. This is the attribution part of the analysis.

Developing a project charter requires a few activities as follows: Identification of stakeholders/customers: A project manager needs to identify all the stakeholders and the customers at the start of the project. Identification of project scope: Scoping is a critical activity to create a boundary of what work to be done and what not to be done. We will cover the project scope in detail in the Scope Management chapter. Identification of project risks: Identifying risks is an on-going activity of the project manager. This activity starts from the onset of the project and continues throughout the life-cycle of the project. Identification of project assumptions: There are multiple activities which are done on the basis of organizational and project environmental factors. These activities are executed either because of process policies or certain assumptions. Thus, it is important to uncover all the assumptions. This is usually done by the project manager in discussion with other stakeholders. Identification of high-level project requirements and objectives: Detailed level requirements and objectives are not possible at this stage of the project. Hence, the Project Manager should focus on high-level project requirements and objectives. Identification of project success criteria: Project Manager should also identify the success criteria of the project. This is used as a baseline and compared with actual project performance. Documentation of identified elements: All the identified elements need to be documented to help standardize the project work.

The project charter includes the following: Business Case Project Selection Project Purpose or Justification Measurable Project Objectives and related success criteria High-level Requirements Assumptions and constraints High-level Project description and boundaries High-level risks Summary milestone Schedule Summary Budget Stakeholder list Project approval requirements (i.e, What constitutes project success, who decides the project is successful, and who signs off on the project) Assigned Project manager, responsibility, and authority level Name and authority of the sponsor or another person (s) authorizing the project charter

charter should provide, either directly or by reference, including: Requirements Business needs Summary schedule Assumptions and constraints Business case, including return on investment

Several respondents described that measures of such activity were available either through their EHR vendor or their health system. However, it was not clear the extent to which the measures captured the time spent directly interacting with in-basket management and the related activities users complete to support their in-basket management (for example, navigating to the medication list or lab results before returning to the in-basket to complete the message). Although this may be possible to capture algorithmically, it nonetheless adds assumptions to the interpretation of the measure.

Although not explicitly addressed by respondents, burden associated with in-basket management is likely more common among the following clinicians: those who place orders; those who work on large teams (and thus have the potential to be included on messages from a larger number of teammates); those who supervise trainees (and thus have more orders and notes to cosign); and those with large panels or a large number of complex patients.

One practicing clinician and EHR medical director explained, “If we’re being asked to do more than we’re humanly able to do, then it’s going to be difficult to feel good about it at any point.” However, several respondents noted that EHRs may have contributed to the creation of regulatory burden by fostering the perception that discrete fields and alerts about potential safety events could be smoothly built into the EHR, but few people understood how burdensome these additional requirements could be, especially in the aggregate. Nonetheless, several sources of burden associated with EHR use arose that may be valuable to measure using audit log data.

Upgrades to vendor products could affect audit log data and challenge interpretation. In particular, upgrades could add or remove steps the end user needs to take in the EHR to complete a particular task; these steps would be reflected in the audit log data but would need to be mapped back to existing analytic frameworks to reflect, for example, that entering information in a newly available field was in fact updating the medication list and thus should be considered clinical documentation and not administrative.

Respondents strongly cautioned that audit log data were difficult to interpret, with one EHR vendor representative likening the “truly raw data” to “drinking from a fire hose.”

A vendor respondent explained that EHRs captured different elements of data in the course of clinical use, differentiating between: the highest aggregate level of log data that captures access to records (for example, who logs in and when); the middle level that captures transactions (for example, adding information to a patient note); and the most granular level that captures mouse movement, scrolling, and the module or screen with which people interact.

Several standards and regulations govern audit log data and provide some indication of potential similarities in this data across EHR vendors. However, EHRs produce multiple system-generated logs and transaction-level records, not all of which are audits of user activity.

To address these methodological concerns, researchers have begun to explore computational ethnography—“a family of computational methods that leverages computer or sensor-based technologies to unobtrusively or nearly unobtrusively record end users’ routine, in situ activities in health or healthcare related domains”

Website and mobile analytics: If a heuristic evaluation provides you with qualitative data, then analytics tools will provide the necessary quantitative information you need. Most people should be familiar with the basic functions of Google Analytics, such as traffic source, traffic flows and trends over time; more advanced functions can elucidate user flows within the website, conversion (and abandonment) hotspots and what users are doing before and after they visit your site. Tools such as Kissmetrics and Crazy Egg can supplement basic analytics with features such as heat maps and churn rates; app analytics can be collected either through Google mobile analytics, are through a dedicated tool such as Mixpanel. Make sure you are going far back enough in the analytics to recognise trends, rather than basing the audit on isolated data points.

The most popular post-task questionnaires are: ASQ: After Scenario Questionnaire (3 questions) NASA-TLX: NASA’s task load index is a measure of mental effort (5 questions) SMEQ: Subjective Mental Effort Questionnaire (1 question) UME: Usability Magnitude Estimation (1 question) SEQ: Single Ease Question (1 question)

The ISO 9241-11 standard defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”.

Naturally, the questions that come to mind are …“Which metric shall I use?”, “How shall I obtain the components needed to calculate it?”, “Is this metric reliable enough to give a realistic picture of the degree to which my system is usable (or not)?”.

I use the System Usability Scale (SUS) to measure user satisfaction with Web sites. This satisfaction scale has been around for several years, and is used by many usability testers. SUS scores can range from 0 (very little satisfaction) to 100 (very high satisfaction).

Efficiency is usually measured by determining how quickly users are able to complete scenarios, test questions, or tasks. It is possible that users are able to successfully complete an activity, but they take "forever" to do it. In fact, they may take so long to complete a web-based task that they find it easier to call someone on the telephone or walk to someone else’s desk and simply ask them.

determining how many scenarios, test questions, tasks or other types of activities that participants are able to complete.

measure effectiveness, efficiency, and satisfaction–they all measure different aspects of the usability of a Web site. If only one or two of these measures are used, it would provide an incomplete or partial picture of the possible human performance and user satisfaction results.

It is important to realize that usability is not a single, one-dimensional property of a product, system, or user interface. ‘Usability’ is a combination of factors including: Intuitive design: a nearly effortless understanding of the architecture and navigation of the site Ease of learning: how fast a user who has never seen the user interface before can accomplish basic tasks Efficiency of use: How fast an experienced user can accomplish tasks Memorability: after visiting the site, if a user can remember enough to use it effectively in future visits Error frequency and severity: how often users make errors while using the system, how serious the errors are, and how users recover from the errors Subjective satisfaction: If the user likes using the system

Fig. 1 illustrates the range of topics that can be informed by research using EHR audit log data, using the health care quality domains put forth by the Institute of Medicine (now the National Academy of Medicine) (1) to organize the topics.

A third common use of audit logs was to study care team structure and dynamics (17 articles).58–64,74,96–104

Most articles used audit logs to study EHR use directly (62 articles, see Table 4 for details by article).20–81 This included how often providers accessed individual pieces of information,20–32,78 patterns of EHR use across features,33–48,62,69–76 and total duration of EHR use.49–55,63–68 More recently, studies began to use audit logs to examine clinical workflows extending beyond the EHR, using audit log timestamps to mark clinical event boundaries (34 articles).64–97

all EHRs in the United States now track at least 4 pieces of information about every episode of patient record access including who accessed which patient record at what time and the action they performed in that record such as adding, deleting, or copying information

Starting in 2005, the Security Rule of the Health Insurance Portability and Accountability Act (HIPAA) required all healthcare organizations to “implement hardware, software, and/or procedural mechanisms that record and examine activity in information systems that contain or use electronic protected health information.”8 The second stage of the Meaningful Use regulations,9 released in 2014, further clarified that certified EHRs must maintain audit logs adhering to the ASTM E2147 standard for tracking health information technology (HIT) use.

Per the protocol, the investigators were blinded to the physicians’ identities, so physician demographic data were reconciled by a third-party honest broker to avoid participant identification in the study dataset available to the investigative team.

Vendor-derived EHR-use platforms compile EHR audit log data to synthesize information on physician time on EHR activities for practice leaders. The primary objective of this study was to derive and report the 7 proposed core EHR use metrics across 2 healthcare systems with different EHR vendor product installations in a cross-sectional analysis.

Inclusion of all physicians using currently available vendor-derived EHR use data would have artificially inflated normalized metrics by including excess inpatient EHR activity and/or EHR activity related to care of patients scheduled to be seen by trainees.

for every 8 hours of scheduled clinical time, these physicians spent more than 5 hours on the EHR. Of this time, on average, approximately 33% of this time is spent on documentation, 13% on inbox activity, and 12% on orders (Figure 2).

in their current form, EHR audit logs and vendor-derived EHR use platform data have multiple limitations to allow derivation of all proposed core EHR metrics and comparison of metrics across vendor products. Five of the 7 proposed core EHR metrics were measurable, while 2 were not. Even for the metrics that were measurable, some of the measure implementations were imperfect or differed substantially between vendors.

For Yale-New Haven Health, Epic Signal was the data source for EHR use data (selected over UAL Lite and Event Log due to its more active data capture of EHR activities), and Epic Clarity was the source for scheduling data.

Observational analyses of big databases, however, are not the preferred choice for comparative effectiveness research. We resort to observational analyses of existing data because the randomized trial that would answer our causal question—the target trial—is not feasible, ethical, and timely.

teaching EBM in medical school has been investigated in various studies. Smith et al [13] conducted a controlled trial to look into the effectiveness of EBM courses for residents. Another recent study conducted by Nasr et al [14] evaluated four EBM workshops taught to residents-in-training and postgraduates in medical school. Interestingly, Slawson et al [15] suggested the importance of information management in the teaching of EBM back in 2005, and with the rapid development of big data, better computer processors, and the maturation of machine learning in recent years, information management is more important than ever in EBM.

Published online 2020 Jan 30

JMIR Res Protoc.

Sources of RWE data can come from electronic health records (EHRs), health surveys, claims and billing data, product and disease registries, mobile health apps, and personal smart devices.

Remarkably, even in the well-studied field of cardiology, only 19 percent of published guidelines are based on randomized controlled trials. 13 Furthermore, because evidence from randomized controlled trials may not generalize well in many clinical situations, physicians are forced to rely on their best judgment instead of on quantitative analysis.

One definition of evidence-based medicine is “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.”

Shortliffe et al.9 recently highlighted the six capabilities a system must possess in order to support clinical decisions including transparency, rapid turnaround, ease of use, the relevance of answer, respect for users, and solid scientific footing.

While novices may have more broad general information needs and may struggle with applying evidence-based practices, they would rather require guidance on applying existing knowledge to practice. On the other hand, Data Consult Service would be beneficial to clinicians who routinely apply evidence-based medicine practices, identify gaps in current evidence, and deal with complex clinical cases.

outpatient care tends to be more influenced by non-clinical factors, such as socio-economic determinants or patient compliance and adherence. When prescribing treatment or diagnostic procedures, healthcare provider should account for patients’ capacity to pay for their treatment and diagnosis, compliance with medications, and follow-up appointments and other factors.

We have discovered that the need for clinical evidence does not diminish with physicians’ maturity: while residents and practitioners at the early stages of their career35 may experience lack of clinical expertise and knowledge, more senior physicians have already gained this expertise and oftentimes require more complicated and precise evidence. This observation relates to the Dreyfus model of skill acquisition,36 according to which one progresses through five levels of proficiency: novice, advanced beginner, competent, proficient, and expert. At the expert level, clinicians no longer have to rely on the set of rules or analytical principles that they had been taught but use their background experience to “intuitively” make decisions. On the other hand, experts can identify knowledge gaps and acknowledge their relevance to clinical scenarios. Constant search for evidence when no established practice exists seems to be a feature of experts, who tend to apply analytical problem solving to novel or complicated clinical cases.

On contrary, the length of inpatient stay might give physicians more time to search for evidence. They also tend to face more challenging cases, which oftentimes requires more thoughtful research, team collaboration, and experience sharing. This may imply that inpatient physicians are more likely to practice evidence-based medicine and use additional knowledge sources to find additional evidence for their clinical cases.

The main obstacles to applying evidence in practice were low patient compliance and socio-economic determinants along with mistrust of clinical guidelines and clinical trials. While other studies22,31–34 listed lack of time, personal unawareness, disengagement, and passivity as the strongest barriers, our findings just partially support it. Indeed, outpatient physicians could rarely find time for literature review and usually thought of their clinical scenarios as routine and straightforward.

Clinicians reported that the current studies appeared to fail to provide evidence for newly marketed drugs (Should a diabetic patient on ACE inhibitors, diuretics, and SGL2 inhibitors be taken off diuretics as SGL2 inhibitors act as diuretics?). The available evidence also was said to inadequately cover certain populations such as pregnant women, children, elderly, and patients with multiple chronic conditions and rare disorders.

The open-ended semi-structured questionnaire included questions about perceived information needs and knowledge resources that physicians use to fulfill these needs (textbooks, electronic resources such as PubMed and commercial tools, clinical consultants, and pharmacists). During 1-h in-depth face-to-face interviews, we asked interviewees to provide the number of questions for which they found no appropriate or insufficient medical evidence, the time expended searching for evidence, and examples of the questions that occurred. We provided examples of possible questions from our practice as well as other participants’ scenarios. To facilitate recall, we provided scenarios related to different aspects of care (diagnosis of rare events or disorders, treatment strategies, quality of care, patient compliance).

there is little knowledge on evidence utilization and sufficiency among specialty physicians or senior physicians working in secondary and tertiary care services.

Smith7 in 1996 summarized the studies related to doctors’ information needs, concluding that the prevailing part of the unmet information needs consists of treatment questions that are often complex and highly patient-specific. This finding was supported by Ely et al,4,8,9 who found that most of the immediate questions generated during consultations remain unanswered, mainly due to the lack of time.

population, intervention, comparator, outcome, and timeframe (PICOT)

Matching is a way to identify subsets of patients who are similar in most respects other than in the treatment they received, in order to reduce the chance that observed differences in outcomes are caused by variation in properties other than treatment but that also impact the outcome (commonly referred to as confounding).

The Workflow for Fulfilling a Consultation Request

The personnel costs for our geographic area (San Francisco Bay area of California) for this team are estimated at $505,000 per year. Yearly data access infrastructure, cloud compute, licensing, and professional service expenses come to an additional $70,000 per year. With these assumptions, the cost of running such a service would be approximately $550 per consultation.

Over the course of the 18-month pilot study of the first 100 consultations, the median consultation turnaround time was 5 days. As the team gained experience and the service workflow matured, by the end of the study, reports were returned within 48 hours. The average time devoted to each report ranged from 3.75 hours to 5.75 hours.

Evidence derived from RCTs, however, often does not generalize to the majority of patients, who tend to have multiple comorbidities, take many medications, and differ from individuals enrolled in RCTs on many characteristics,4 resulting in an inferential gap between the evidence that is available and that which is needed.5,6

Here, we discuss the methods that are used for data extraction, processing, and analysis in our consult.

About 90% of claim denials are preventable, and effective prevention of claim denial can result in more than $5 million in additional revenue for an average hospital, according to Becker’s Hospital Review.

Claims submission is reliant on the accurate input of preregistration, charge capture and medical coding data.

Average hospitals experience about a 10% claim denial rate, according to Becker’s Hospital Review.

steps of the revenue cycle may include the following.

one of the core components of health information management (HIM), which also covers electronic health records (EHR) and patient privacy management methodologies.

In 2016, Addison26 found a single feature that is correlated with BP, called the slope transit time (STT) that requires only a single PPG signal. The STT reflects the steep trend of rising pulse wave.

Since the volume and distension of the arteries can be related to the pressure in the arteries, the PPG signal produces pulse waveforms that are very similar to pressure waveforms generated by tonometry; however, PPG offers the added advantage that it can be measured continuously using miniature, inexpensive, and wearable optical electronics.

The 10-minute hack gives you a really simple goal (work on this one thing for 10 minutes without stopping). The side benefits of doing this successfully are that resisting the urge to heed distractions helps strengthen your prefrontal cortex, and that once you’re actually 10 minutes into working on your task, it’s easier to continue working on it.

it’s important to get mental rest to keep your prefrontal cortex working well, and to set aside special time dedicated only to prioritization

Love your prefrontal cortex: 

each time you perform a thought or action, you make it easier for your brain to reproduce that thought or action. This has some pretty clear implications for positive vs. negative thoughts, and explains why meditation is so powerful (which I’ll get to in a bit).

What kind of work do I need to do right now? Is there anything extremely pressing, or can I let my mental state guide the work that I decide to do right now? What kind of mental state am I in right now? Am I in the mood for draft writing, outlining, researching, exploring, or polishing? (Throughout the process, I began to codify the different types of work required to produce my writing.) Is there something I can do to get myself into the right mental state? Over time, I realized there were different “hacks” or rituals that would help me switch mental states. Exercising, massages, different types of music, different types of teas, epsom salt baths, and neurotransmitter-supporting amino acids all eventually served their own purposes. I also had different venues to do different types of work: for example, a cafe in a skyscraper high above the city was better for higher-level brainstorming, while a dark, small room in the public library was better for polishing or research. (Which is consistent with academic findings.)

the associations we observed between stress and window switching, inbox work duration, and inbox work outside work hours do not necessarily prove that the latter factors cause stress. It is possible that physicians who are busier during work hours have more stress and also make more window switches, have more inbox work, and have to do more inbox work outside work hours.

Our finding that the window switching rate was positively associated with stress could reflect the complexity and repetitiveness of physicians’ EHR interactions, as indicated in prior work [Soh JY, Jung SH, Cha WC, Kang M, Chang DK, Jung J, et al. Variability in Doctors' Usage Paths of Mobile Electronic Health Records Across Specialties: Comprehensive Analysis of Log Data. JMIR Mhealth Uhealth 2019 Jan 17;7(1):e12041 [FREE Full text] [CrossRef] [Medline]56], and the efficiency issues often associated with physicians’ satisfaction with EHRs [Williams DC, Warren RW, Ebeling M, Andrews AL, Teufel Ii RJ. Physician Use of Electronic Health Records: Survey Study Assessing Factors Associated With Provider Reported Satisfaction and Perceived Patient Impact. JMIR Med Inform 2019 Apr 04;7(2):e10949 [FREE Full text] [CrossRef] [Medline]57].

Higher rates of EHR window switching, longer inbox work duration, and a higher proportion of inbox work done outside of work hours were associated with higher stress. Daily stress patterns showed 3 waves of stress: in the first hour of work, at or after lunch hours, and in the evening.

Time spent on inbox work during work hours was positively associated with stress (P<.001), whereas time spent on other EHR activities during work hours was negatively (but very weakly) associated with stress (P<.001). Inbox work outside of work hours was positively associated with stress during work hours (P<.001). Interestingly, the proportion of inbox time spent on patient messages was not associated with stress. Surprisingly, batching inbox work for the day was also positively associated with stress (P<.001). Finally, days of the week were predictive of stress, with Mondays and Thursdays negatively associated with stress, whereas Tuesdays and Wednesdays positively associated with stress (P<.001 for each).

3 temporal patterns of work, with a silhouette score of 0.41, indicating moderate separation between these clusters (ie, distinct groupings). Figure 1 shows the average hourly time spent in the inbox and other EHR work (such as charting and order entry) for physicians in each cluster. Group 1 (n=10) represented physicians who spent time in the inbox outside work hours, in the evenings and early mornings; group 2 (n=17) represented physicians who worked mostly within work hours; and group 3 (n=20) represented physicians who spent some time on inbox work after hours that were mostly contiguous to work hours.

days with inbox work batching as days where 70% or more of the total inbox work duration occurred in 3 separate blocks of time or less.

We used the Gaussian Mixture Models clustering algorithm [Reynolds D. Gaussian Mixture Models. In: Li S.Z., Jain A.K. (eds) Encyclopedia of Biometrics. Boston, MA: Springer, , MA; 2015.46] to find distinct patterns of inbox work. Features in the model included the distribution of inbox time in work hours and outside of work hours contiguous and noncontiguous to work hours. Multiple feature and cluster counts were tested, and the clustering that yielded more balanced clusters and had a reasonable silhouette score (a score that indicates how distinct or overlapping the clusters are) [Starczewski A, Krzyżak A. Performance Evaluation of the Silhouette Index. In: Artificial Intelligence and Soft Computing. ICAISC 2015. Lecture Notes in Computer Science, vol 9120. Cham: Springer International Publishing; 2015.47] was selected.

We created hourly time bins and variables from the log data to quantify how time was attributed to different activities and different types of inbox messages per hour. These variables, which were collected for every hour, included the number of minutes spent in the EHR, the number of minutes spent in the inbox, the number of minutes spent working on each inbox message type, the number of tasks performed, and the number of window switches (ie, clicking a new computer window).

An example of an ANSI accredited SDO is HL7, and example of an HL7 developed standard is FHIR (Fast health Care Interoperability Resources.

Quality matters more than quantity. If you read one book a month but fully appreciate and absorb it, you’ll be better off than someone who skims half the library without paying attention. Speed-reading is bullshit. Getting the rough gist and absorbing the lessons are two different things. Confuse them at your peril. Book summary services miss the point. A lot of companies charge ridiculous prices for access to vague summaries bearing only the faintest resemblance to anything in the book. Summaries can be a useful jumping-off point to explore your curiosity, but you cannot learn from them the way you can from the original text.* Fancy apps and tools are not needed. A notebook, index cards, and a pen will do just fine. We shouldn’t read stuff we find boring. Life is far too short. Finishing the book is optional. You should start a lot of books and only finish a few of them.

Many of the studies cited in this work 1,2,4–6 classify EHR events by topics such as chart review and notes. These classifications are generated by human experts. It is possible that experts could create an ontology for the classification of all codes. However, the classification of certain codes may be context dependent. Computational methods like the work here are useful to infer the latent EHR tasks based on audit log sequences.

We learn EHR tasks by identifying which events are contextually similar and then clustering into groups of similar events that we refer to as tasks. Contextually similar events are those that tend to be used when the preceding and following events are similar. In order to identify contextually similar events, we calculate vector representations of each event using the Word2Vec algorithm11,12. In this algorithm, each event is coded as a feature and is used to predict the surrounding events, and this is referred to as the context window. We set the output dimension of the Word2Vec algorithm for each event to 300 as in Lyu et al13.We cluster events into tasks based on vector similarity using the k means algorithm in the sklearn package14 in Python version 2.715. In order to constrain to a manageable number of tasks, we pre-specified the number of clusters to five in the k means algorithm. The event logs are segmented by EHR task clusters and visualized in ProM Process Mining Software version 6.916 using the inductive miner algorithm17. The input of ProM is events and sub-sessions within a task. The sub-session is all events that correspond to a task within a session. A process tree is generated for each cluster using the inductive visual miner algorithm. Process trees are a directed hierarchical graph in which each node is an event with children events that are subsequent in the sub-session sequence. There are multiple behaviors for children in the process tree. A sequence is used when all of the children of a node are executed in order. A concurrency indicates that all of the children are executed, but independently of one another. The process trees are visualized with traces or paths that traverse the edges from the beginning to end of the graph. Edges and nodes are weighted by sub-session counts.

To identify sessions, audit logs are sorted by provider, patient, and timestamp. A new session begins when there is a change in provider or patient, or there is greater than 5 minutes between timestamps10.

The audit log data includes a patient ID, employee ID, event ID, event description, and a timestamp for the start of each event. There is not a clear indication of when each event ends.

The EHR audit log contains a ledger of each activity that takes place when a patient record is open. Examples of activities include: view, edit, accept, cancel, print, and exit. Activities take place within EHR modules (e.g., notes, diagnoses, vital signs, etc.). In general, each activity and module combination is represented by a code. When a clinician enters the vital signs module, updates vital signs, and then exits the module; view, edit, accept, and exit activity codes will be recorded in the audit log with a corresponding timestamp9.

ProM process mining software

To address the variation in patient needs, we focus our analysis on newborn inpatients with an inpatient stay less than or equal to three days. Newborns were selected for this study, because in this population care patterns may be more standardized. In addition, we restrict to those with length of stay less than or equal to three days to reduce the likelihood that newborns with complex care are included in the study8.

Past studies of physician EHR behavior leverage interviews or observation1,2,4–6, making them difficult to scale. In contrast, we attempt to learn provider behavior in the EHR from audit log data. Leveraging the EHR audit logs allows for an automated analysis of provider workflows7. The audit log data are a reflection of provider activity in the EHR and, are impacted by variation in provider behavior and patient needs

Understanding of EHR tasks has the potential to reduce burnout. When EHR tasks are learned, time spent on tasks can be quantified, from there bottlenecks can be identified, and interventions can be made to increase the efficiency of workflows.

For example, in outpatient settings, a lower completion rate of routine health maintenance events such as cancer screening or vaccine administration represents missed opportunities in clinical care. Although both patient- and clinician-specific factors contribute to these missed opportunities, these process metrics can be considered as potential downstream effects of burnout.

One such example is wrong-patient medication errors. Such errors can be detected using EHR-based metrics of ordering activity33,58 and have been previously used to evaluate patient safety outcomes.59–61

The Maslach Burnout Inventory survey,50,51 which is often used for burnout measurement, has a sensitivity of 1-year (questions follow the pattern, “in the past one year”). For measuring proximal impacts, such as clinical workload, long-term measurements are unlikely to be true reflections of burnout associated with their immediate workload. Instead, newer scales, such as the Stanford Professional Fulfillment Index (PFI),52 which has a 2-week sensitivity, can be a suitable alternative.

The role of ecological momentary assessments, an ambulatory data collection technique that queries present-moment experiences, allowing for real-time sampling of thoughts, feelings and behaviors, can provide more stable estimates of situated behaviors (eg, stress, task load).48

First, as shown in Figure 1B, measurements need to account for the “individual characteristics” that contribute to burnout—including demographic and personal characteristics and physiological and behavioral activities. Towards this end, newer mobile and wearable devices afford the ability to capture behavioral patterns (eg, physical activity, sleep, and fatigue) that are contributors to the emotional demands of clinicians

Adopting complementary measurements

compared to instruments such as the NASA Task Load Index (NASA-TLX) scale for creating reliable metrics for standardizing and validating EHR-based audit log measures as a proxy for cognitive load.

ognitive burden can be determined based on the nature of EHR-based clinical activities, including number of tasks being performed during a time period and order of these tasks; number of task switches within a patient chart, where increased task switching denotes greater expenditure of cognitive resources or switch cost (eg, from documentation to medication ordering); and number of task switches between different patients, with even higher switch costs (ie, multitasking).

Although these standardized measurements provide a foundation for research, these can only be used for developing descriptive characterizations of activities performed by clinicians (eg, message volume, frequency of out-of-office work). To associate clinical work activities with actual or perceived workload, these measures should be converted to meaningful correlates of workload. One potential approach—especially within the context of clinical workload—is to utilize these measures to compute metrics of cognitive load experienced by clinicians.

CONSIDERATIONS FOR APPLYING THE CONCEPTUAL FRAMEWORK

In other words, these activities can be used to capture work-related factors (ie, clinical workload) contributing to burnout by addressing questions such as: (a) what types of activities were conducted, (b) how much time was spent on these activities, (c) what was the sequence of activities performed, and (d) where the activities were conducted (eg, work, home).

A CONCEPTUAL FRAMEWORK FOR EVALUATING BURNOUT

Our focus in this perspective is on highlighting the conceptual considerations needed to augment the use of EHR-based audit logs beyond characterizing clinical workload by incorporating personal and situational factors that contribute to burnout. Towards this end, we describe (a) a conceptual framework for utilizing EHR-based audit logs for evaluating burnout within the situated context of a clinical environment, and (b) pragmatic considerations for applying this framework using existing tools and instruments for investigating burnout and its impact on downstream patient safety and quality outcomes.

Burnout is a work-related syndrome involving 3 dimensions: emotional exhaustion, depersonalization, and a sense of low personal accomplishment.1–3 Although burnout has been reported to be prevalent in nearly 50% of physicians,1 a recent systematic review found that prevalence estimates among physicians range from 0%–80.5%, highlighting variations in the definitions of burnout and its assessment methods.

This led to the unexpected and potentially useful finding that inbox work duration was inversely associated with the mean number of work segments per message, even after controlling for multiple other factors. This finding signifies that work patterns that involve opening a message multiple times do not necessarily cause inefficiency.

Our finding that working at a medical center with high or low mean inbox work duration was independently associated with an individual physician’s inbox work duration suggests that some medical centers may have adopted processes that enhance the efficiency of inbox work for all PCPs in those locations.

Primary care physicians in this study made nearly 80 attention switches involving the electronic inbox on an average workday. Message quantity was a key correlate of both attention switching and inbox work duration. High inbox work duration was also associated with having a higher percentage of panel patients initiating messages and with working at a medical center with a high mean PCP inbox work duration. These findings suggest that interventions to assist PCPs with message quantity might help reduce both attention switching and inbox work duration.

Context and Interpretation

Major Findings

Discussion

This study’s cross-sectional design could not elucidate whether more attention switching caused longer inbox work duration or whether having a longer duration of inbox work simply created more opportunities for attention switching. To address this, we created a variable that reflected a physician’s propensity to complete each message in one sitting, formulated to be independent of the amount of time spent in the inbox. This variable was the number of work segments per unique message, where a segment was a discrete period of work on a message. For example, a message that was opened on 3 separate occasions resulted in 3 segments. We hypothesized that physicians who opened a single message multiple times would have higher attention switching and higher inbox work duration.

Attention switching was defined as switching from one work context to another. A PCP doing inbox work switching to other EHR work was counted as a switch from inbox to other EHR work. If a PCP was inactive on the EHR and then opened an inbox message, this was counted as a switch from non-EHR work to inbox work.

Other EHR work included all actions not occurring after inbox message access, such as reviewing patient medical records, entering orders, or writing medical record notes.

For most actions, if 45 seconds or more elapsed with no EHR activity prior to an action, the preceding period was classified as non-EHR time. This criterion was based on operational reporting standards and our review of the frequency distributions of time gaps in preliminary data. In selected situations, we used a higher criterion to capture actions that often required more than 45 seconds. For example, the action of sending a reply to a patient’s message was often preceded by more than 45 seconds while the PCP typed the reply. As the Epic access log detects only mouse clicks initiating or completing an action, it did not record time spent typing the reply. Thus, for the time period prior to sending a reply to a patient’s message, we allowed up to 120 seconds, enabling us to capture and count most periods spent typing messages.

Inbox work periods started when a PCP opened an inbox message and ended when the PCP completed activities associated with inbox messages and switched to another activity. For example, opening a patient’s secure message followed by a review of the patient’s medical record, placing an order, and returning to the inbox to reply to the patient was not counted as an attention switch, because all actions occurred in the context of processing the patient’s secure message

This study was approved by the Kaiser Permanente Northern California Institutional Review Board with a waiver of informed consent. This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

Methods

Our aims were to describe PCPs’ frequency of attention switching associated with electronic inbox work, identify potentially modifiable factors associated with attention switching and inbox work duration, and compare the relative association of attention switching and other factors with inbox work duration.

Future evolution of this approach should include the use of standardized metrics7 that can be applied across vendors and that can be used for cross-organization and cross-specialty comparisons, as well as preworkflow and postworkflow and policy interventions.

Similarly, at the inaugural Patient, Practitioner, and the Computer conference held at Brown University in 2017, physicians from among the 6 other industrialized nations present (Canada, United Kingdom, Denmark, Portugal, Israel, and Australia) responded with puzzlement to the degree of their American colleagues’ distress about EHRs. The conference report concluded, “The United States experience was contrasted with those of other nations, many of which have prioritized patient-care documentation rather than billing requirements and experienced high user satisfaction.”