Delving into icon testing
General info about icons:
- Resemblance Icons — depicting a physical object like an envelope for mail
- Reference Icons — more conceptual like clamp for file compression
- Arbitrary Icons — some are universal (rare) like hamburger menu. If not, eg. “?” for help
And if its too conceptual, I add text on the bottom to help comprehension. (Making an icon link/button is another kettle of fish!)
I didn’t feel that our current icon of an ATM from our design set looked like an ATM. So I wanted to do some tests, though I had no budget from the project to pay for UX research. Which might involve a 1000 user click test online.
So I asked our UX researcher to give me some pointers so I can do some quick tests.
The four criteria for testing an icon from the Nielsen Norman website are:
- Findability: Can people find the icon on the page?
- Recognition: Do people understand what the icon represents?
- Information scent: Can users correctly guess what will happen once they interact with the icon?
- Attractiveness: Is the icon aesthetically pleasing?
The ATM icon is classed as a Reference Icon and I want to test it’s Recognition.
I didn’t test the icon within its context or the page design, but searched for different concepts of “ATM” on the web and placed them on a strip with other icons.
I printed these strips of paper out and went downstairs to ask some people who weren’t designers to tell me what each icon is, no right or wrong answer.
A success would be if they identified the one icon in each strip as an ATM, a failure would be if they didn’t get the ATM icon. I asked 13 people and each person I showed the 6 strips, one at a time. Here are the results:
Only one person got ATM (1/13) with our current design, while the last two had high success(11/13) (12/13). If they didn’t get ATM, they got “machine that gives me money”. I think the last design is better than the second last, as some people thought it was a bin, throwing rubbish.
I noticed that after they got the ATM on two/three strips, they thought its like a game, like which one is like the ATM? As they saw a running theme.
Also some questions I had for our UX researcher:
- People were looking up to me and ask if they got it right
My response: “I’m not going to tell u if its right or wrong, just keep going”
[UX researcher] Yep, that is a reasonable response. You could set the expectations and scene at the start of your discussion with them.
Essentially saying that the purpose of this is not being right, its simply to work out if people in general, when they see these icons on a screen with no one to help them, know what they are.
2) People asked if they want me to say what it looks like or what they think the intention is, like intention is ‘time’ vs ‘clock’, or like ‘umbrella’ vs ‘cover’.
My response: “Whatever comes first in your mind” (because I only cared about the ATM icon).
[UX researcher] You want to know if they recognize the ATM so you can just say “Just let me know what it is rather than the intention.”
3) If they struggled with an icon, not knowing what it is.
My response: “Skip if you don’t know, you have a time limit of 5 seconds for each one”.
[UX researcher] Just ask for their best guess, you don’t want them to skip the ATM one.
4) They wait or respond when I scribble notes — I guess they notice I don’t scribble notes on most of the icons.
My response: I don’t know if this is a problem, but it seems to affect them but I didn’t change anything.
[UX researcher] A nice trick I do is just take notes for all of them, or at least pretend to for all then they don’t know which one you are interested in.
Round 2
After the first round, the researcher advised showing one strip to one person only, which meant I had to ask more people. This was to avoid that game mentality when they knew they were searching the the ATM icon and skew the results.
There were 6 strips of paper and I wanted results of 7 people, so I ended up asking 42 people showing them one strip! Here are the results:
It seems like concept 6 is the winner, and concept 5 - which was our current ATM icon, kinda flunked. Concept 2 had a high success rate, but someone also thought it is a bin. These findings were similar to the first round.
So in two hours with no budget I could make an argument to change the ATM icon and redraw it with money spitting out concept with buttons on the side.
However, there were biases that affected the results:
- I just pulled icons from the net without redrawing it in our brand style and testing those. Because once you redraw in the brand style it might be unrecognisable again. This is something I would do in the future before testing it.
- Sizing of icons — when I printed it on paper it is bigger than it might be on screen or the intended size of the icon. So the ATM icon is meant to be 32px by 32px, so when I printed it out it was bigger and so it looks different. Something to be aware of.
- The placement of the icon in the row can affect the comprehension, like putting a monitor before the ATM can make the person think the ATM icon is a computer — bias in positioning and other icons affecting its meaning.
All in all, this wasn’t a comprehensive test, for example it wasn’t a paid test to the UX researchers who could conduct a 1000 click test for comprehension.
But its about leveraging time, money and resources.
So for two hours with biases there is still some merit to the whole test, like maybe 60% of the result compared to a paid click test of 85%.
