Identify the chief complaint and pertinent past medical history

t would be difficult to assign, or even expect, a single “correct” diagnosis at this stage, the emphasis being on considering the most appropriate set of diagnoses (“high frequency, highly plausible, high impact”).

suggestions were not weighted based on how reasonable or appropriate they were

searches the underlying knowledge base in response to clinical features input in free text and displays diseases with matching textual patterns arranged by body system rather than in order of clinical probability. Thus, ISABEL functions primarily as a reminder tool to suggest 8 to 10 diagnostic hypotheses to clinicians, rather than acting as an “oracle.

the quality of a diagnostic hypotheses plan—irrespective of whose efforts it represented (system or user).

In this setting, it was not essential that the system possessed a high degree of diagnostic accuracy, so long as its suggestions positively influenced users' diagnostic reasoning

More sophisticated measures of system performance proposed by Berner et al.14,22 also studied the ranking of diagnostic hypotheses in a system's list and other discrete indicators of diagnostic quality, such as relevance and comprehensiveness, generated by comparing the DDSS diagnostic hypothesis set to a “gold standard” set generated by expert clinicians.

Early studies expected the DDSS to be able to predict the “correct” diagnosis in a diagnostic dilemma.

ew studies have been able to convincingly show changes in physician behavior or improved patient outcomes

during diagnostic decision making (DDSS)

In an early project, we integrated,revised, and expanded the ten heuristics by Nielsen [54] and the eightgolden rules by Shneiderman [56] to form 14 principles customized forthe health domain [57].

Enter HPI (History of Present illness).2.Enter PMI (Present Medical Illness).3.Document Social History.4.Document Family History.5.Enter Vital Signs.6.Enter Order Consult.7.Document Coding of the Procedures.8.Entering the Lab Order.9.Document Instructions – Other Therapies.10.Order Radiology Study.11.Document Comments in A/P Diagnosis.12.Review Coding of Medical Encounter.13.Document Follow-up Plan.14.Associate Orders/Medication/Labs.

In this study, we used the Keystroke Level Modeling (KLM) to estimate time on task, task steps, and mental effort for fourteen prototypical use cases for the AHLTA EHR.

There are many other types of representation analysis, some of which are being developed and evaluated in our EHR Usability Lab at the National Center for Cognitive Informatics and Decision Making in Healthcare.

This method starts with the identification of the system hierarchy of an EHR system. The system hierarchy was created by visually inspecting the user interface items from top to bottom and left to right.

he Designer Model has 60 functions and it was obtained through a complete system walkthrough. The User Model has 80 functions and it was developed by conducting interviews and surveys with end users. The Activity Model has 97 functions and it was developed by doing a field study that involved many sessions of shadowing and observation (for details, see [45]) of the end users in the clinics.

Fig. 2. A conceptual model of function discrepancies (from Chen, 2008 [45]).

n this example, the Goal is “treating high glucose level in a pre-diabetic patient”; the operation is “writing a medication prescription”, the objects for this operation include patient name, doctor’s name, diagnosis, medication name, dosage, frequency, duration, route, etc.; the constraints include the dependency relations between operation and objects (e.g., operation “write a medication prescription” and the objects “Metformin” and “500 mg”), between objects (e.g., the object “glucose level” and the object “Metformin”), and between operations (e.g., the operation “write a prescription” and the operation “modify problem list”)

Operations are performed on the objects under the constraints to achieve the goals.

it tells us the inherent complexity of the work, it separates work context (physical, social, organizational, etc.) from the inherent nature of the work; and it supports identification of overhead activities that are non-essential for the work but introduced solely due to the way the system is implemented.

The ontology of work domain is the basic structure of the work that the system together with its human users will perform.

User analysis is the process of identifying the types of users and the characteristics of each type of users. For the EHR domain, types of users include physicians at various levels (e.g., attending, fellow, resident, medical student, etc.) and in various specialty areas (family practice, intensive care, dermatology, surgery, etc.), nurses at various specializations, medical technicians, medical staff, patients and family members, and so on.

The essence of usability is the representation effect. Representation effect is the phenomenon that different representations of a common abstract structure (e.g., a work domain ontology, see Section 3.2.1 for details) can generate dramatically different representational efficiencies, task difficulties, and behavioral outcomes

Successful applications should be both useful and usable, and they should be considered together because they are not independent, as demonstrated by Butler et al. [35]

Landauer argued that successful applications should be not only usable, their functionality should also be useful

Satisfaction under TURF is similar to satisfaction under ISO definition of usability. Under TURF, satisfaction refers to the subjective impression of how useful, usable, and likable the system is to a user.

Task steps refer to the number of steps (both mental steps such as recalling a drug name from memory and physical steps such as clicking a button on the screen) needed to complete a task

Under TURF, a system is usable if it is easy to learn, efficient to use, and error-tolerant. How usable a system is can be measured by learnability, efficiency, and error tolerance.

The evaluation was performed by three independent evaluators whose results were integrated into a master list of all violations. Then each evaluator independently rated each violation for its severity on a scale of 1 to 4 (1 = cosmetic; 2 = minor; 3 = major; 4 = catastrophic) and their ratings were then averaged, as shown in Fig. 9.

In an early project, we integrated, revised, and expanded the ten heuristics by Nielsen [54] and the eight golden rules by Shneiderman [56] to form fourteen principles customized for the health domain [57]. We have since applied the fourteen principles to a variety of healthcare domains [57], [58], [59], [60].

1.High affordance: Operation can be perceived by using external cues in the interface.2.Medium affordance: Operation can be perceived by external cues in the interface and internal knowledge of the application.3.Low affordance: Operation can be perceived mainly by using internal knowledge of the application.

ISO defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.”

useful, usable, and satisfying

Designing and implementing EHR is not so much an IT project as a human project about usability, workflow, patient safety, and organizational change [8], [11], [18], [22], [23], [24], [25], [26], [27]

usability issues, which did not receive significant attention in the EHR community until recently

However, there are huge gaps between the status quo and the potential of EHR, primarily due to cognitive, financial, security/privacy, technological, social/cultural, and workforce challenges [8], [9], [10], [11].

You didn’t need to write down your opinions about anything; you were able to vent your spleen in the post-session debriefing with the researcher, and that was recorded on video tape so it could be transcribed and analyzed at some later stage.

After all, if SUS indicated that people didn’t rate a system very highly on usability, we had video tapes of the sessions they spent interacting with the system to go to

But why is there the rigmarole around converting the scores to be between 0 and 4, then multiplying everything by 2.5? This was a marketing strategy within DEC, rather than anything scientific. Project managers, product managers, and engineers were more likely to understand a scale that went from 0 to 100 than one that went from 10 to 50, and the important thing was to be able to grab their attention in the short space of time they were likely to spend thinking about usability, without having to go into a detailed explanation. (Also, where differences in perceived usability were achieved, having a scale of 0 to 100 was likely to make the differences be perceived by team members as being greater than on a smaller scale—not that it makes any difference when it came to actual analysis.)

he normative data collected by Bangor, Kortum, and Miller (2008) and Sauro (2011) provided the basis for positioning SUS scores as percentiles, providing a more meaningful basis for interpreting SUS scores

Tullis and Stetson’s (2004) research showed that using SUS enables you to get a measure of the perceived usability of a system with a small sample (say, 8-12 users) and be fairly confident that you’ve got a good assessment of how people see your system or product. As Figure 2 shows, using SUS means that you reach a “correct” conclusion quicker, and that you reach a greater level of consistency between respondents sooner than you do with other questionnaires.

SUS is not diagnostic. That is, it does not tell you what makes a system usable or not.

SUS scores have a modest correlation with task performance, but it is not surprising that people’s subjective assessments may not be consistent with whether or not they were successful using a system. Subjective assessments of usability are only one component of the overall construct of usability.

You might want to decide, for example, whether it would be better to use a website or a voice-based system to access a bank account. SUS allows you to make that comparison, at least as far as perceived usability goes. Because SUS is pretty much technology-neutral, you can continue to use it as technology evolves over the years, and you don’t have to continually reinvent questionnaires.

Because of the efforts of many researchers over the years, it’s now possible to choose SUS and be confident it is a valid and reliable measuring tool, be able to make a comparison between the scores you achieve and some normative standards (and thus with other systems or products), and to have some idea not only of whether people like your system, but also whether they would recommend it to others.

n my previous research, to do with the design of information displays and decision support systems, especially where they were concerned with the support of fault diagnosis for operators of continuous industrial processes (e.g., Brooke & Duncan, 1981, 1983) or for decision making by doctors (Sheldon, Brooke, & Rector, 1985)

Eventually, about a decade after I first created it, I contributed a chapter describing SUS to a book on usability engineering in industry (Brooke, 1996). Since then its use has increased exponentially. It has now been cited in (at the time of writing) more than 1,200 publications and has probably been used in many more evaluations that have not been published. It has been incorporated into commercial usability evaluation toolkits such as Morae, and I have recently seen several publications refer to it as an “industry standard”—although it has never been through any formal standardization process.