- Jul 2024
-
arxiv.org arxiv.org
-
suggestions at different levels of abstraction
Providing instructors with different levels of abstraction of summary, ranging from high level summaries and evidence and explanation regarding such summaries.
-
information from interactions with externalsources within the work domain
In LMS this means considering holistic student performance instead of specific instances of student performance.
-
gives me clues as to why I should pick one over the other
Justification of why AI made certain suggestions should be transparent to humans.
-
citing a lack of comprehension for AI-generated code translation and “spotting errors in‘foreign’ code” as challenges.
I wonder if similar challenges can be found in LLM powered data analysis. Here the problem would be the ease of verification and spotting inaccuracies in AI's reason behind decision making.
-
hen using the new tools in practice, many users, such as programmers,report increased cognitive load, frustration, and time spent on the tasks that GenAI is intended to support.
This aligns with our findings, where inaccurate information generated by AI will indeed reduce the productivity of instructors since they will need to determine whether the information provided is accurate before making a decision.
-
- Jun 2024
-
arxiv.org arxiv.org
-
It is common sense that AI would hardly be perfect, especially in knowledge-rich domains, such as usability testing.Thus, it would be inappropriate to have the WoZ AI suggest all the ground-truth problems. To make the WoZ AI morerealistic, we randomly added 5 (18%) false problems (false positives) and removed 4 (14%) true problems (false negatives)in the two selected videos in total.
Probably important justify the inaccurate nature of AI, that motivates this study of trust and verification
-
mixed design
For our study, might also need to use a mixed design approach to mitigate learning effect.
-
nine UX evaluators were recruited to independently review the videos, identify segments in thevideos where users encountered problems, and write descriptions for the problems. Next, two other UX researchersreviewed the identified problems and their descriptions, discussed and consolidated a final list of ground-truth problemsand descriptions. In total, there were 20 UX problems in the coffee machine video, and 8 in the website video.
Can be used to identify ground truth problems in our study
-