Oct. 2021 - Dec. 2021
Zoom, Google Workspace
Cameron Dudzisz,
Eesha Parasnis,
Yuxin Tao, and myself
Moderated usability tests; synthesized data; constructed forms and documents
Pratt Institute Libraries’ goal is to support the Institute’s academic programs by providing resources and services to students, faculty, and staff. The Libraries intend to upgrade their discovery interface, and want to assess whether the new interface is usable and helpful for students.
Our team of four UX researchers conducted nine moderated remote usability tests to evaluate the usability of the new discovery interface and proposed recommendations to improve the search experience for undergraduate students.
Our client wanted feedback on the usability of the new discovery interface, specifically for the ask-a-Librarian chat, keyword searches, filters, PDF readers, and icons from undergraduate students at Pratt Institute.
We used the following research questions to guide our study:
Are students able to find and access what they are looking for?
What are students' impressions of the new search interface?
Keeping our research questions in mind, we conducted a usability study to evaluate the discovery interface.
Usability Study
Findings
Recommendations
A screener was sent via email to students at Pratt Institute, and 71 undergraduate students qualified for the study. A smaller pool was invited to participate, and care was taken to invite students from a variety of grade levels and majors.
Ultimately, nine undergraduate students were selected to complete the usability tests.
Our tasks simulated a typical student’s journey through the new discovery interface. We asked participants to locate the LibraryChat feature, search for an ebook through the general search bar, download a chapter of the ebook, and create an APA citation of the ebook.
Prior to starting our usability tests, four pilot tests were conducted to ensure that the tasks were easy to understand and not too difficult to complete.
Scenario: You are writing a report on art styles during the Renaissance era. You are using the Pratt Libraries website to conduct your research.
Task 1: You are having trouble getting started with your research. Which tool would you use to connect with someone to find help?
Task 2: You remember your professor gave you some articles to read about art during the Renaissance period and you want to look up related books. Find a book that fits these criteria which you can read on your computer, and skim through the chapter titles of the book.
Task 3: You want to start reading on your subway commute. Save Chapter 1 on your computer.
Task 4: You want to include this book in your report as an APA reference. How would you find the information you need?
A rainbow sheet was utilized to synthesize participant observations from the usability tests. Our team members inputted observations from the usability tests they moderated and denoted which participants exhibited these behaviors.
Observations that were seen across four or more participants were prioritized, examined in further detail, and used to inform recommendations.
Overall, participants liked the new discovery interface. Most participants believed that this new interface was an improvement compared to the existing platform. In particular, participants praised how fast and quick the tool was to use as well as the minimalist and simple design.
Despite the general approval and noted refinements, we also identified some areas where further improvements are needed.
was the average System Usability Scale score, indicating a higher than average level of usability
was the most common word participants used to describe their search experience
of participants needed assistance to complete the second task
of participants had trouble accessing the new interface's PDF Reader
The Library Chat button in the bottom right corner is too small and some participants missed the icon entirely. In addition, four participants initially attempted to locate this tool on the left sidebar.
Increasing the size of the Library Chat icon and adding a secondary link to it in the left sidebar would make the tool easily discoverable, allowing it to better assist students during their research.
Participants were unsure what the "Access now" submenu in each source card was for, and had difficulties accessing the PDF Reader within that submenu.
Adding a separate PDF Reader button on the source card and source details page would simplify the process and allow students to quickly navigate to the PDF Reader.
Only sources from Pratt's internal collection can be accessed via the PDF Reader. However, participants had trouble discerning which sources were included in Pratt's collection, and were often redirected to external sources instead.
Adding some distinguishing text on each source card that provides information about whether a source is hosted internally or externally would make this clearer to students and streamline their research process.
Almost half of the participants applied filters when conducting searches, and participants had mixed feelings towards the filters. Participants were confused by the wording of some categories such as "Content Provider."
Displaying tooltips when hovering over a filter category would be a very helpful to help students understand what each filter does, thus eliminatinating confusion or ambiguity during the research process.
My team and I delivered our report and presentation to our client and he was very pleased with our work. He thought that the Recommendations 2 and 4 in particular were great suggestions to consider adding in the discovery interface, which Pratt Institute Libraries is looking to implement in the summer of 2022.
Our client also asked us about our personal experiences with the new interface and how it compared to the current library website. I mentioned how I appreciate having the search suggestions list in the new interface, as this feature is missing from the current website. Our client was very glad to hear this feedback, and he hopes to implement the search suggestions feature next summer as well.
I enjoyed having the opportunity to see how students compared their experience on the new interface to the previous one; most of my prior projects have concerned evaluating an interface without comparing it to a previous iteration, so I appreciate being able to see which interface participants preferred and why.
If I were to redo this project, I would consider rephrasing our second task to be more specific, especially since only sources in Pratt's collection can be viewed in the PDF Reader. Most participants needed assistance during this task, so specifying that the book they are looking for should be part of Pratt's collection could be a way to reduce ambiguity and moderator intervention during the test.
