Think aloud protocol has long been one of the most commonly used evaluation
techniques for usability testers, and is also used by visualization researchers,
both to evaluate tools and methods as well as to investigate the formulation of insights.
But, think aloud user studies are a costly, time intensive process, which up until
now are almost always run synchronously, either in-lab or via video conferencing.
In fact, in a survey of visualization papers we conducted, only 1 of 67 think-aloud
studies were conducted asynchronously, meaning without a researcher present.
In this paper, we make two primary contributions. The first is a technical contribution called
CrowdAloud, a tool for conducting and analyzing asynchronous think aloud studies. The second is an
investigation of the feasibility of crowdsourced think aloud studies. We conduct two separate user
studies, one which compares crowdsourced think aloud to in-lab thinkaloud, and another comparing
think aloud responses to more traditional text responses. In those studies, we found
that think-aloud is comparable to in-lab studies, and produces more insights than text responses,
with a similar quality. This video, however, will focus on the first contribution, Crowdaloud.
CrowdAloud is built on top of reVISit, an existing tool for conducting user studies,
but one without any audio capabilities. reVISit utilizes a domain specific language,
or DSL, for defining tasks and study structure.
CrowdAloud can be enabled in that DSL with one line of code,
and which tasks should be recorded in the study can also be set in the DSL.
Optionally, you can add a mic-check task,
which will stop participants until they have demonstrated they have a functioning microphone.
After a study is complete, CrowdAloud can
be used to conduct a qualitative analysis on the think aloud data.
CrowdAloud has 2 primary views, the replay view, and the tagging & transcription view,
which are intended to be used together. First, let's talk about the tagging & transcription view.
Recorded audio from the study is automatically transcribed,
and researchers can edit the transcription at will.
Researchers can also create codes, each with their own name and color.
After creating a codebook, each audio snippet can be assigned codes. Codes are
specific to the logged in user, allowing multiple users to independently code.
The true strength of CrowdAloud comes when researchers utilize provenance data in their
study. Using provenance, CrowdAloud can recreate the state of the original study,
replaying the study similar to a video. The provenance data is synced with the audio data,
so that researchers can always see the same stimulus that participants saw while listening
to their comments and insights. Here is a real example from one of our crowdsourced participants.
The replay view is also synced with the tagging view, even if they are
in separate tabs, allowing researchers to use multiple monitors for analysis.
To aid with analysis, we also have an overview table view. The table view includes basic
information for each participant in a task, like time spent, interaction count and word count. You
can also view tags which were assigned for this specific task. Additionally, researchers can
search for specific tags, in this case to find all participants who made a UI/UX comment, question,
or suggestion. From here, we can jump directly to their task analysis, to see what they said.
We hope that our findings, as well as our tool CrowdAloud, encourages
the visualization community to conduct more crowdsourced think-aloud studies in the future