Monday September 21st, 2009 by Björn Balazs
This summer I participated in the SoU project with Gallery, the open source photo sharing software. The broad goal of the project was to conduct a survey to learn more about Gallery’s users in moving toward the release of the 3rd version of the software. The project was a success in terms of garnering useful data for Gallery, but, importantly, it was an incredibly valuable learning experience for me. As a student working towards a masters in Human-Computer Interaction, I have learned about and conducted user research and usability studies in my coursework. But, what SoU provided me that no school project ever could was the opportunity to engage in research that has an actual impact on improving a product— and the opportunity to experience all the challenges that go along with that.
At the start of the project, I have to say I was a bit overwhelmed with what direction to take, but with the guidance of Björn as my mentor, the help of Jakob, a former SoU student who continues to work on the Gallery project, and the awesome tools of UserWeave.net, I soon found my way. Here are some of the lessons I learned along the way.
1. Two (or more) surveys are better than one
One of the first stumbling blocks for me was how to design this survey in a way that really captured information about Gallery’s users in a useful way. It took a while to get my bearings on what would be the most useful areas to probe. Where would the most interesting, pertinent information be found? I thought of dozens of questions I could pursue, but how would I know what the best answer options would be? And how could I be sure they would result in any sort of interesting data? With a little guidance from my mentor, the idea of a pre-survey questionnaire was born. Doing a small-level survey with lots of open ended answers exposed me to the possibilities- and also showed other question areas that should be pursued. Doing the pre-survey activities helped me to hone in on the areas that needed more treatment and refinement, and also showed where I was going in the right direction. And most importantly, I learned that not every question or area needs to be addressed at one time. Rather than trying to make a big survey be perfect the first time around, it’s better to just try something out and learn from mistakes.
2. Sometimes you have to reframe your focus to reach your goals
The goal of the survey was to learn more about Gallery users. The challenge was to do that in a way where we could learn about what differentiates those who choose Gallery from those who choose to use other products or share photos another way. And, we wanted to find out more not just about Gallery users, but about potential users as well. In order to do that, we had to do something that seemed strange on the face of it– remove focus from Gallery in the survey. In fact, only one question even mentioned Gallery at all. Going into the project, I never would have imagined that, but after the fact, I’m convinced it allowed us a better perspective from which to see why people choose Gallery, or why they do not. Only by putting the focus squarely on the motivations, tasks, and desires of everyone who shares photos online were we able to understand who Gallery users truly are.
3. Breaking the rules a little can be a good thing
One of the things I’ve always thought to be a best practice in conducting large-scale surveys is to limit open-ended answers, if not exclude them altogether. There are many reasons for this, not the least of which is the resulting thick, dense data that’s hard to tease apart. From the beginning, I sought to eliminate or severely limit free answer questions. In fact, one of the purposes of the pre-survey questionnaire was to identify the range of possibilities for multiple choice and relevance analysis questions, so we wouldn’t even have to have “Other” as an open-ended option. As I began to develop the actual user survey, I limited the free-answer options, eventually removing them all. When Jakob asked why, I explained the challenges and pitfalls of open-ended data. But, he countered with a convincing argument about how useful this thick data would be for the folks at Gallery. So, we broke the rules (or guidelines, I suppose) and did a few free answer questions anyway, which, in the end, provided incredibly valuable data and ended up being one of the most useful parts of the survey. And, thanks to the awesome filtering tools of UserWeave.net, it was easier to tease apart the data that I had imagined. While there was no formal synthesis and analysis of the free answer data, it enabled us to see more in-depth what certain segments of users were thinking, and to see trends in various segments to a certain extent.
These were just some of the lessons I learned with the project. And hopefully there will be many more. Perhaps the best part of SoU is that its gotten me involved in the Free/Libre/Open Source Software world. Not only was this project an amazing learning experience, it proved to me that I can make valuable contributions to the open source community. I’m already on to the next user research activities….