Some thoughts on testing icons

Icons are used to visualise a term. An icon test evaluates the strength of this relationship based on subjective and objective indicators.

It is really great to see how many of you already took part in our KMail-Icon-Test. A lot of questions arrived my by mail or on this blog, so I thought I would just explain a little about testing icons…

We do this testing, because icons are useful and beautiful – they save place and people can recognize them faster than they can read text. Practically icons work via a visual-metaphor. If that metaphor, however, is not understood by the user, the icon does not work out. One can easily realize, that it will be much easier to find a visual metaphor for “save” than for “save as” or even more complex or abstract computer actions.

With the tests we mainly want to answer two questions:

  • We only have limited resources. Now we want to work on some new icons – but which icons should we work on?
  • We want KDE SC to be usefull for everyone in the world. So, do our icons and our terms work for everyone in the world?

The icon test finds indicators for answering these questions, by asking you – the user – to allocate icons and terms. We can measure how well icons and terms match. And we can also do this individually for any language the study runs in (so we get the world answer). Technically we have found a lot of indicators for the quality of icons, that focus on explicit or ambiguous allocations of icons, missing icons and time spent to actually decide for an icon. All these indicators help us to answer the above questions.

So this is a method mainly for evaluation and no not for finding inspiration. But still, this is not absolutely true. Some people complained about not offering a “no icon fits”-button. There are two things to say about this:

  1. The evaluative result is the same – whether we offer the “no icon fits”-button or not – and whether people read instruction to skip a term or not. To assure this we us a multiple indicators approach.
  2. The inspirational result is worse when we offer an explicit “no icon fits”-button, because we do not get the idea about approaches to the visualisation of that term. Typically you do not get a random result if a term has no well corresponding icon. You do get some icons that drop out and you can try to understand why. So you might just find ideas about how to construct the metaphor for that missing icon.

So we will continue  to explain that a term can be skipped, but we will not offer a “no icon fits”-button.

There have also been some complains about the terms, missing context and the sense of life. Well, it’s not always sunshine. Testing typically means not being “real”. In other words: Testing always has some restrictions. With the icon test we focus on the association between the icon and a corresponding term. If the term is rubbish, the icon has no chance. As terms for our KDE SC icon tests we use the labels next to the icons or tool-tips of the icons (if no label is around) you can find in the application. We tell the participants that they can find these icons in the context of a certain application – this time KMail.

And, as I mentioned above, we can compare the results of different groups. May it be the spanish speaking people with the Windows(TM)-only users. Thus we can thoroughly investigate the relationship between icon and term. And we think the indicators we get are much better than no indicators, and we are well aware of the existing restrictions. You can find a sample result of an icon-test here.