Have in mind the genres tested.

Looking from a perspective of "how might we use such tools in UX" we're better served by looking at documents that UX generates through the lens of identifying parallels to the study's findings for business documents.

To use AI to generate drafts, we'll want to look at AI tools built into design tools UXers use to create drafts. Those tools are under development but still developing.

Agreed, much of user testing focuses on the experience of the user when using the tool and depends upon users speaking to voluntarily making the quality of output part of their evaluation of the tool.

A possible counter: UX research professionals are not subject matter experts in the work deliverables generated in using the tool and not equipped to assess the quality of those deliverables. But that doesn't address the need the author calls out. Solving this will depend on the domain in which the tool is used and connecting with SMEs that can evaluate output in that domain.

In educational technology, we measure the quality of output by measuring the growth in students' mastery against state standards as they use the tool. This is why reporting has become the key feature for EdTech products.

The numbers for how users divided their time may not be reliable as they're self-reported.

Still leaves me curious about the accuracy of reported brainstorming time.

This points to AI providing the best time savings in draft generation, which fits with the idea of having the AI generate the drafts based on the professional's queries.

For UX designers, this points to AI in a design tool being most useful when it generates drafts (sketches) that the designer then revises. Where UX deliverables don't compare easily to written deliverables is the contextual factors that influence the design, like style guides or design systems. Design too AI assistants don't yet factor those in, though it seems likely it will, if provided style guides and design systems in a format it can read.

Given a draft of sufficient quality that it doesn't require longer to revise than a draft the designer would create on their own, getting additional time to refine sounds great.

I'm not sure what to make of the reduced time to brainstorm when using AI. Without additional information, it's hard not to assume that the AI tool may be influencing the direction of brainstorming as professionals think through the queries they'll use to get the AI to generate the most useful draft possible.

Mistaken human assumptions that AI will generate what a human would given the same prompt are reinforced by claims by those selling AI tools that such tools "understand human language." We don't actually know that AI understands, just that it provides a result that we can interpret as understanding (with the help of our cognitive biases).

This claim to understanding is especially misleading for neural network-based AI. We don't know how neural networks think. With older Lisp based AI we could at least trace through the code to see how the AI thinks.

By providing interfaces to AI tools that help us duplicate the aligning, clarifying, and iterating behaviors that we perform with human collaborators we can increase the sense that users can predict what results the AI will provide in subsequent iterations. This will remove the frustration of working with a collaborator that doesn't understand you.

Collaborating with humans beats collaborating with AI because we can sync up our mental models, clarify ambiguity, and iterate.

Current AI tools are limited in the methods they make available to perform these tasks.

Natural language alone is a poor interface for creating an effective prompt. So bad that communities and businesses are surfacing to help people create effective prompts.