Feb. 2023 – May 2023
Miro, Google Workspace, Figma, Google Analytics, Hotjar, Tobii Pro Eye Tracker
Sandy Leegumjorn, Tess Porter, Anyelina Wu, and myself
Moderated tests, synthesized post-test questionnaire and SUS data, and developed 3rd finding and recommendation
Smarthistory is an informational website that provides free educational art history resources to students, teachers, and art history enthusiasts. As the art history discipline has become less Western-centric within the last few years, Smarthistory wants to understand how they can organize their content in a more flexible and robust manner.
Our team of researchers examined behavioral analytics and conducted eye tracking tests to understand how we can make it as easy as possible for mobile users to find what they are looking for.
After meeting with Smarthistory, we brainstormed the following questions to guide our study:
How do users navigate to find what they are looking for?
What are the characteristics of a typical user visit?
What are the main pathways users take to find the information they need?
How effectively and intuitively do pages inform users of existing content?
From there, we evaluated behavioral analytics and conducted eye tracking tests to provide recommendations for improving the information architecture of the mobile website.
Behavioral Analytics
Eye Tracking
Findings
Recommendations
We analyzed Smarthistory's Google Analytics data from February 14th, 2022 to February 14th, 2023 and reviewed metrics for mobile traffic to understand how users are accessing and interacting with the website.
Furthermore, we reviewed Hotjar data from February 9th to February 28th, 2023 and studied heatmaps and recordings of mobile sessions to investigate user behaviors on the website.
From our analysis, we found that content pages, which include essays, videos, images, and additional resources for artworks, are the most visited pages. However, landing pages, which provide introductions to art histories through geography and time periods, are not frequently visited.
We discovered that most users arrive on the website via search engines, direct links, and referral sites, but 81.5% of users visit only 1 page and 81.5% visit for less than 10 seconds.
With the first task, we wanted to understand how users navigated to specific artworks from the homepage, and their organic interactions with navigation elements such as the hamburger menu, search bar, and bottom navigation.
With the second and third tasks, we wanted to learn about users' initial reactions to content pages and how they navigated between them, as mobile users tend to visit content pages most.
For the final two tasks, we were curious about how users navigated to different topics within the same art history section, and how they might interact with landing pages.
Scenario: You are a student preparing to write a paper on Leonardo da Vinci’s works and his impact on future art movements.
Task 1: Find where a copy of the Mona Lisa was found.
Task 2: Next, find when da Vinci returned to France.
Task 3: Next, find an artistic technique that characterizes European Neoclassicism in the 1700s (the 18th century).
Scenario: You are working on a project about Modern art (before 1980).
Task 4: Within the Modernisms 1900-1980 section, find the name of one influential photographer.
Task 5: Find a definition of Dada Readymades.
We used the Tobii Pro eye tracker to record our participants' eye movements and capture where they were looking as they completed each task.
Following each test, we used retrospective think aloud to hear our participants' thoughts on their experience. We showed them gaze replays of their eye movements and asked them about where their eyes fixated on certain points, navigation choices, and interactions with the content.
To sum up their experience, we asked participants to complete a post-test questionnaire as well as a System Usability Scale assessment.
We learned that 75% of our users did not think the website was easy to use and 88% did not feel confident using it. Furthermore, 75% thought that they needed to learn a lot of things before they could get going with this website.
In addition, participants were asked to describe their experience on Smarthistory in the post-test questionnaire. The most common words were "informative" and "text-heavy" and we were surprised that one participant had written "user-hostile."
My team and I synthesized our data using a rainbow sheet to find problems that occurred across multiple participants. While participants thought the information on the website was trustworthy and liked the bottom navigation, they faced issues with navigation and finding information.
From our rainbow sheet, we chose four problems that we deemed most severe to address further in our findings and recommendations.
When users landed on the homepage, three out of eight users did not understand the purpose of Smarthistory and what it offers.
On content pages, users were overwhelmed by the amount of text on the screen and expected to see a summary of key points at the top of the page.
“I was expecting it to tell me basic information about the Mona Lisa, but I didn't see it even though I was reading carefully.” - Participant
Users extensively scrolled through the hamburger menu but as the menu contains both art history geography and time periods, users had trouble finding what they were looking for.
Similarly, it is difficult for users to locate specific works of art from the landing pages. Only 25% of users used the Beginner's Guide at the top of each landing page whereas the majority scrolled past.
The search bar auto-searches before users finish typing their queries, which hinders their experience finding specific artworks or pages.
“It can search automatically but it shouldn’t take away what I was typing.” - Participant
In addition, users weren’t able to immediately locate the search bar on the homepage because the search icon is not easily noticeable.
We also found that users are unable to tap outside the hamburger menu to exit it, and users mistapped on the bottom navigation because the touch targets are too small.
Users responded positively to the bottom navigation, with 75% of users thinking it is easy to use and 50% of users expecting to find something like it during their visit.
“Especially when stuff is interconnected (...) it makes it very intuitive and helpful.” - Participant
While 63% of participants used the bottom navigation during the test, only 12.5% (one user) thought it was easy to find.
Adding an introductory paragraph to the homepage would help users learn more about Smarthistory and its offerings.
Similarly, adding a table of contents with key points to the beginning of content pages would allow users to jump to specific sections and find information easily.
Grouping subsections by world regions and time periods within the hamburger menu would create shorter lists that reflect the categories users expect to see.
Making sections collapsible on the landing pages would reduce scrolling. Moreover, adding a search bar, pagination, and filters will allow users to quickly find desired articles.
Allowing users to continue typing during auto-search will vastly improve their search experience.
Confirming whether “SEARCH” appears next to the search icon on all mobile screens would make the search function more discoverable.
Giving users the option to exit the hamburger menu by tapping outside of it will align with standard conventions.
Increasing all touch targets to at least 44 x 44 pixels, especially for the bottom navigation, will reduce mistaps.
50% of users suggested moving the bottom navigation to the top of the page to make it easily discoverable.
This navigation bar includes a breadcrumb trail to the subsections the article is located in. Tapping “Europe 1300-1800” opens the full menu.
Increases in pageviews, session duration, and pages/session during A/B tests will indicate the new navigation bar is successful.
We delivered our presentation to Smarthistory and they were appreciative of our work as they gained a better understanding of what mobile users need to find information easily and effectively. They were keen on learning more about our eye tracking tasks, methods, and the System Usability Scale, and were particularly invested in fixing the auto-search issue I discussed.
"The search glitch... we gotta fix that!" - Smarthistory Digital Strategist
This was my first time using eye tracking in a usability test and I think eye trackers are incredible tools that add extra layers of depth to evaluation studies. I found it insightful to observe where participants were looking during the test and to use retrospective think aloud to hear their thought process. I like that it felt more like a conversation about their experience than using think aloud in a traditional moderated usability test.
I was also excited to work with a mobile interface, as it was a good challenge to learn how to use the Tobii Pro eye tracker on a mobile website. I enjoyed comparing the mobile experience to the desktop experience and looking for areas where improving an element on one device would improve it on all screens.
If I were to redo this project, I would want to use a collaborative analysis tool such as Dovetail to consolidate my team's findings, test materials, and recordings. We used Google Workspace to organize our files, but since we had so much data, I think our analysis would have been more streamlined if we were able to easily tag and organize our findings and documents.
