What We Learned About User Testing Will Shock You
Tricks to decide if you need a User Lab or Guerrilla Test
UX Researchers’ community is living its own Barcelona vs. Real Madrid when you ask them: “What’s your team? User Lab or Guerrilla Test? Like football conversations, these methods generate controversy and many hours of conversation. Definitely, when you need to choose a test method, you will find some rivalries.
Like Barcelona and Real Madrid, both methods are very good. At Opentrends we know that because even building a User Lab, we still do Guerrilla Testing with customers.
If you don’t know which method choose for your next test, we show you some tricks to take the right decision in your next research challenges.
Perhaps the best definition of guerrilla testing was coined by Martin Belam. He defined guerilla testing as “the art of pouncing on lone people in cafes and public spaces, and quickly filming them whilst they use a website for a couple of minutes.”
Basically, guerrilla testing means going into a coffee shop or another public place to ask people about your prototype. It’s low cost and relatively simple testing that enables real user feedback.
But does Guerrilla testing can be conducted in the nearest public place (cafe, sports venue, shop, etc)? That’s not always true. You need to pick a location where your target audience spends their time. For instance, if you’re testing a new mobile app for a retail chain, you might go to one of the stores and test there.
A Test Lab is a room where you evaluate in detail every single user action with specific tasks and exercises. The Lab has an adjacent room called Observer Room to analyze through a mirror (and in real time) these actions. Like in Police Stations, users are suspicious people to interrogate. Test Labs usually have technology and software to measure accions and find UX/UI mistakes in any product or digital service. At Opentrends we’ve experimented with eye tracking tests and sensors to understand crazy things: user emotions. Thus, we discover those processes where users aren’t aware of what are they doing.
Usability Test Labs are an heritage from marketing research. During years, marketers have performanced focus groups in rooms following observation techniques and using artifacts as the interrogation room we said before.
The lab, as a controlled environment for performing user testing is more artificial and the insights we get are less spontaneous than Guerrilla Test, where you can’t take up more than 30 minutes of their lives. But in a controlled environment like a Lab, you can do other Research activities and Design Thinking exercices.
Context is essential in User Testing. That’s why we put it at the top of the list only in case you don’t want to read the full article. If you choose a User Lab to do user testing, people will probably go there with a clear goal in their minds: “don’t be dumb”. They maybe will dress different as they do, will talk different as they usually do and also will behave different as they do. They will preconfigure themselves such in a way that their actions or answers won’t be focused on the test itself.
Guerrilla Test, on the other hand, gives us two critical points: test in a real environment and catch users unprotected. If you test a flights’ app, test it in an airport line. If you test a food delivering app, test it during an office lunch time...
Some UX experts suggest guerrilla testing can be done with almost anything, including concepts drawn on paper. As a result, many researchers print out designs and ask test participant to test paper sheets. This isn’t right way to do guerrilla testing because that’s not how someone experiences a product in real life. You can’t expect people to understand how your product works by flipping through paper pages.
While you don’t need to have a finished product to conduct guerrilla testing, you still need a prototype testers can experience. The more it resembles a real product, the more valuable feedback you’ll receive.
Election seems to be easy but there is a key factor before taking the final decision: the “fantastic” legal and provacy terms. Getting permissions and acceptances for Guerrilla Usability Tests is not always easy.
If you can’t get permissions for your Guerrilla Text on the ground, you should go to a Lab and customize it to set the most realistic context.
This is not a perfect scenario, but if you want to create a simulated context, focus on disturbing things like sounds (available online) to simulate people talking, planes taking off or ringing mobiles. Also “cheat” on users by creating an the way they’d feel: pressure them by setting up time specific tasks; invite them to do combined tasks to prevent focusing in one thing only… By reproducing environments, you will have the chance to find new needs. Pero esto requerirá un extra en la preparación del test.
Both decisions could work but always have one thing in common that may bother users and get them defensive: the signature of permission for taking pictures or/and record voice and screen. This situation could be uncomfortable for some participants. Instead of doing that, try to take notes and leave users act on their own.
If context is important, test preparation is one of the success (or failure) keys. Guess having an amazing context but fail during the preparation. Avoid big fails like this. Both tests, Guerilla or User Lab could be as difficult to prepare as reliable on its results.
For example, if you go out finding users you don't need to set up rooms, devices, software tools and other settings. But you’ll need to be a hunter and catch as many users as you need on the go. Otherwise, your Guerrilla Test will fail.
In a Test Lab, everything should be scheduled and ready for users and testers. Maybe it is artificial but it’s amazing if you don’t like improvisation. If everything is well prepared, you won’t lose time between tasks. User Test Labs are taken from Marketing research and offer safety and controlled experiments with some people observing and recording movements, facial expressions or whatever user looks at. This is something we lived during last Mobile World Congress in Barcelona, where we “eyetracked” everything.
All these gadgets and software can provide amazing insights. Of course, it doesn’t mean Guerrilla Tests don’t give us serious results. But there are some voices in Researchers’ community still believing that aren’t scientific enough. Guerrilla Test detractors think you are invading users spaces, privacy and conformability. They think labs are neutral places and users will find more issues because they will take tasks as a job and finally get a gift card or compensation.
Once compared this two testing methods, maybe you can decide better in your next user test. Both of them have their limitations and strengths. So, which is your team? Street Testers or Laboratory Rats?