Intermediate results of the icon tests: Breeze

With a series of icon tests we currently study effects on the usability of icon design. This article however does not focus on these general design effects but presents findings specific to the Breeze icon set.

Introduction

The introduction of the new Breeze icon set in KDE let us again wonder, what aspects of an icon set actually takes what impact on the usability of it. We investigated Oxygen and Tango Icons for the LibreOffice project before, but our focus then was on checking all icons of the standard tool bar. This time we focus on different icon sets and will use 13 common actions to compare them.

With this series we are going to test at least 10 different free icon sets: Breeze, Oxygen, Tango, Faenza, Nuvola, Nitrux, Elementary, Crystal Project, Humanity and Treepata. These icon sets differ on various aspects: use of color and details, flat or not and sometimes even on the metaphor used.

So, we generally want to analyze effects of icon design on the overall performance of an icon set. Statistics on this issue can obviously only be done after all icon sets have been tested. But with every test, we win some specific insights in strengths and weaknesses of each icon set tested.

In this post we share some findings about the Breeze icon set.

The study was finished by 597 participants (drop-out rate 5%) with an average handling time of 2:17 min.

Results of Breeze icons

Table 1 lists the aggregated quality indicators. They show how well all icons that we used for the test were suited to symbolize the different terms. It has a range from 1 (no fit) to 10 (perfect fit), whereas you would expect values of at least 9 for well represented terms.

Table 1: Quality of the icon set for different terms based on assignment ratio (percentage of missing assignments) and conspicuity (or speed of picking icons).
TermQuality Indicator
Cut5.5
Remove7.5
Save7.5
Link8
Redo8.5
Undo8.5
Open9
Paste9
New9.5
Add10
Print10
Search10
Copy10

Table 2 shows a cross-table with the percentage of false associations. These are terms where the intended icon was not chosen by the users, but some other icon was.

Results_Icontest-Breeze

Table 2: Cross-table of icons and terms with percentage of false associations. The direct match is inverted (1-value, e.g. 0.99 for Add) to obtain comparable data.

 

Discussion

Breeze is the best monochrome and flat icon set we tested so far. However, there is still room for improvement:

  • Copy and Paste are mixed up in a degree comparable to other sets. We really need to work on the metaphors used for them.
  • Using innovative metaphors is not always a good choice. This is proven by the paper cut used for Cut. It has been left out by quite a lot of users and was mixed up with Remove. As participants commented, the red Remove icon is rather associated with Cancel than with deleting an item from a list or the like.
  • Another issue is found for New, which is mixed up with Add. The blank page metaphor obviously is not strong enough.
  • Redo and Undo are both working comparable to other sets, even though they are way more reduced. Both have been left out by users quite often.
  • Quite the same is true for Save. A lot of users leave it out, but the rest can associate it quite well. But perhaps it is a better choice to stick with the floppy disc metaphor.

Considering the innovative and hence unfamiliar design (71% report that they haven’t seen those icons before) the overall performance is already pretty good. If we manage to improve those not yet well working icons, it may well be a good choice for Plasma Next as it fits the general design language.

We are happy to discuss this at Akademy.

We are going to Akademy!

We are going to Akademy!

If you know how to design icons and would like to help us to identify metaphors that work better, please contact us. Also, all raw results are publicly available on our open usability platform UserWeave.

As mentioned before: These results only reflect the internal quality of the Breeze icon set. The final interpretation will be done after all sets have been tested. So stay tuned and please participate in our follow-up tests. And, of course, feel free to discuss these findings with us.