avatarSimon Jones

Summary

The article discusses advanced UX lab techniques focusing on observer engagement, interactive discussions, live communication, and an iterative mindset to maximize learning and development during user testing sessions.

Abstract

The article, titled "Advanced UX lab techniques: Discuss, flex and iterate," emphasizes the importance of engaging not only with users but also with observers during UX lab sessions. It suggests that the time spent in a UX lab should be leveraged to learn from both users and observers equally, advocating for inviting a diverse group of attendees, curating a space for discussion, and capturing the thoughts of observers through post-it notes. The author outlines techniques such as encouraging back-room discussions, live communication between moderators and observers, and adopting an iterative mindset to allow for on-the-fly design updates and re-testing. These methods aim to foster a dynamic and collaborative environment that leads to more insightful and actionable outcomes from UX research.

Opinions

  • The author believes that back-room discussions should be encouraged as they can lead to fruitful discussions and ideas for solutions.
  • They advocate for the importance of capturing observer insights and suggest that these insights are as valuable as those gained from users.
  • The article suggests that flexibility and the ability to adapt during testing sessions, such as through live communication, can lead to more effective and engaging research.
  • The author posits that an iterative approach to testing, where designs are updated and re-tested within the fieldwork period, can significantly accelerate the development process.
  • The author emphasizes the value of the "10:10" workshop as a method for capturing and synthesizing the diverse perspectives of all attendees, ensuring that insights and fixes are prioritized and documented.
  • The author expresses that by integrating these advanced techniques into UX lab sessions, the overall experience becomes more insightful, leading to more robust and user-centric product designs.

Advanced UX lab techniques: Discuss, flex and iterate

How user-testing can be used as an interactive event to develop ideas fast.

This is ‘part two’ of my tips on how to make the most of your UX-lab time. Please have a read of part one, as it introduces the thinking behind these techniques, covering:

1. Focus on observers 2. Keeping the lights on 3. Using post-it’s and wall space

Illustration from ManyPixels

A quick TLDR to recap: The time you spend in a UX lab should be used as a way to learn from users AND observers in equal measure. This will have a huge impact on the analysis you do and the recommendations you make. Invite as many as you can from the core team and further afield. Curate a back-room space to encourage discussion and analysis. Assign post-it-writing tasks for each observer so that you can capture what they are thinking. The more you learn from them the better.

So, now that you have curated your space and prepared your observers for their input. It’s time to think about what to do during the sessions. Here are three techniques to help. I’ve also added a bonus workshop you can run at the end of the day to capture the thinking from everyone in the room. 🙌

4. Discussions & solutions-thinking

In my view, back-room discussions should be encouraged, not discouraged. It is in these moments when some of the most fruitful discussions are had, and if they are not endorsed, they will be lost.

Naturally, the first session will be a focused, silent affair while attendees are keen to watch every detail. But as the day progresses and themes start to emerge, discussion will flow.

Use this as a chance to learn from your attendees, about their unique pressures, thinking and ideas for solutions.

An active back-room, where lights are on and discussion is flowing.

Here are some tips on things to do during and in-between sessions:

  • Take notes both from the participants AND your attendees’ conversations. They are equally important to the final output.
  • If someone says something interesting, engage in the conversation. Is that how you saw it from a research perspective? How does it impact the next phase? Who else would find benefit in knowing this? Which assumption does it support or refute? How should the product change? Why is this idea not possible?
  • At the end of a fruitful discussion point, instruct them to capture it on a post-it. Blue/solutions post-it’s are particularly useful, so encourage those to begin with.
  • Between sessions ask everyone to stick their post-its up in the appropriate place on the wall. Use this time to discuss interesting ones (while adding more notes yourself).
  • If design-updates are discussed, encourage attendees to draw or annotate what they are thinking. It is often difficult to verbally describe a visual solution.

Over the years of trying this ‘open discussion’ technique, a few (valid) concerns have cropped up. Here is my take on them:

How can I take notes AND engage conversation? Yep, this is tough and takes a lot of practice and balance. The best solution is to have an additional note-taker attend, freeing you to engage in the discussion. A good way to do this is to offer note-taking to a colleague (an up-and-coming superstar) who is keen to build experience in these settings. But if the resources don’t stretch to an extra person, the alternative is to just find a balance. It is possible to do both, with practice. For example, in any session there will be lull moments when a new scenario is introduced, or warm-up questions are being asked. Use this time for discussion and go from there.

How can we hear if there is a lot of noise in the back room? This is a valid concern and can become a problem later in the day. If your labs allow it, use headphones to block out some of the discussion, particularly at important moments. Otherwise, sit closer to the speaker. As a last resort, you’ll have to ask for a little quite during important moments, but this may stifle discussion. Again, you will need to find a balance.

I prefer to take notes in Excel or Airtable, not post-its. Is this ok? Also, an important point. I’ve found analysing post-it photos to be highly beneficial, as the photos tend to jog more memories. But I also love the power of some of the modern digital tools (Airtable) in enabling analysis. If you prefer to take notes this way, and can’t find an extra person to do this, then this is OK. Just make sure your attendees still take on the post-it duty. Its important to end the day with the walls covered. We’ve trialled this way a few times and works well.

5. Live Communications & Ad-libs

Live chat is important during sessions. Illustration from ManyPixels.

If your moderators are comfortable, encourage live chat communication with the back-room, allowing additional questions asked on-the-fly.

Many researchers already do this, but I’ve added it to this list as there are a number of benefits:

  • Additional questions are more effective within context of the current discussion, as it can feel disjointed to return to the topic at the end. Participants may also find it difficult to recall a particular moment or thought after moving through other screens or topics.
  • As a moderator, you feel more supported with a line of communication out to a colleague. There will be moments when the stimulus has a glitch, or a section needs to be skipped. Resolutions can then be discussed in the back room and passed to the moderator without much intrusion.
  • It creates a much more engaging interactive environment for stakeholders in the back-room. Some of the best moments I’ve had in testing are when a new question is asked immediately and the response is unexpected. These moments are when strong memories are forged.
  • Finally, it helps to create the feeling that things are fluid and change is OK. This is an important mindset to have during lab-based research, as it is often useful to explore, poke at things and try out new ideas.

How to prepare:

  1. Make sure its not a free-for-all. Have one person (the note-taker) as the funnel for all questioning, so that the moderator is not bombarded from multiple angles. This person needs to strike a balance between insight and interruption, which does take practice.
  2. Allow buffer time within the session to allow for additional questions to come through.
  3. Moderators should use a separate laptop for communication. Set it to silent, but make sure screen time-out doesn’t kick in every 5 mins. We’ve tried mobile chat (e.g. Whatsapp), but messages are easy to miss and can take longer to type.
  4. Moderators should inform participants that they will be in communication with their colleagues. We can’t let them think the moderators are ignoring them.
  5. Moderators/note-takers must be experienced enough to be able to ask non-leading questions on-the-fly. This also takes practice, but it’s important to note that perfection is not the goal here, it is flexibility.

I will caveat that the ability to do this does vary by industry, and by moment within the design life-cycle. In many evaluative or summative studies, for example, discussions guides are often locked down to build numbers. These can be critical to prove or disprove assumptions, so questions must be asked in exactly the same way so they are comparable.

But to me, the default should be this flexible approach unless there is a good reason not to.

We are trying to understand not just behaviour, expectations and attitudes, but WHY people react the way they do. In this fast-paced world, there will be unexpected twists that spark new ideas and new questions. We need to be able to respond immediately to push things forward.

6. Iteration mindset

Encourage iteration of stimulus and discussion scripts. Illustration from UnDraw.

As with point 5, it is important to allow for — encourage, even — changes to stimuli during fieldwork.

A former colleague of mine once suggested that as soon as a design is updated, the insight becomes invalid and more testing is needed. So if an update is made after fieldwork this means pushing ahead with untested updates based on the original insight.

You could, of course, run more research. Many product teams work within a regular testing framework for this very reason. You can never truly solve this riddle, as you can never truly test everything. But you can reduce it’s effect, by encouraging iteration during fieldwork. This enables you to push concept/ideas/designs forward at a much faster rate.

At Ipsos Mori, we routinely ran Rapid Iterative Testing & Evaluation (RITE) weeks, to help teams evaluate and develop products. These consisted of 3 full days of UX Lab sessions (Monday, Wednesday, Friday) with 2 full days of iteration between (Tuesday, Thursday). By building in iteration, you quickly change the mindset from fixed testing to solving, updating and re-testing.

Typically, the first testing day (Monday) would involve small fixes and updating a few typo’s (if it’s prototypes we are looking at). While bigger changes happen after we start to see themes emerging. Typically, the last day barely resembles the first day because our mindset, stimuli and discussion guides have changed so much.

To get this right, though, you need to have a few things in place:

  • All the above 5 points (Observers, Lights on, Wall-space, Discussion and Live Comms).
  • Full engagement from the core project team (on all days).
  • Full engagement from a design function capable of working quickly and familiar with research.
  • At least 1 full day in between testing days, sometimes 2 or 3 (so changes can be made and agreed without your designers burning the midnight oil).
  • Moderators who are comfortable with rough-and-ready updates to questioning.

RITE is a specific stand-alone technique that applies an iterative mindset. But there are many others within Agile and Lean development frameworks. If you’re not ready to commit to a full week, or to go full Agile, then make sure you add a ‘design-gap’ between testing days, or even testing sessions.

If you are testing with higher-fidelity (coded) prototypes or live sites, you may need longer between testing. In this case, consider how you might print or lo-fi-prototype some of the designs instead of coding. This will help to keep things fluid.

Bonus Technique: The “10:10” top-line workshop

I mentioned in the first post, that:

“‘What you learn’, comes not just from your participants, but from the observers watching them. ‘A memorable experience’ comes not just from what your participants say, but from the activities that happen in the back room.”

Well, over the years it became clear that all this extra-participant learning needs to be formally captured. It is not enough to indulge conversations, ask more questions and write more notes. If you have many attendees in the room there is a lot of analytical brainpower that should be put to use. A lot of thinking that needs to be extracted and captured.

The 10:10 workshop will do it.

It takes 30–60 minutes and should be done at the end of the day (We sometimes reduce daily participant numbers just to fit this in).

Here’s how it works:

  1. After the last participant, gather all attendees around an empty wall space and give them each a pile of yellow and blue post-its and a sharpie pen.
  2. For 5 minutes, ask everyone to write 10 Things They Learnt during the day on 10 yellow post-its. More than 10 is OK, but not less. All researchers should take part too.
  3. One-by-one, ask each person to reveal their post-its and place them on the wall under the title “What We Learnt”. Each new topic goes underneath the last. When a post-it is of the same topic, place it to the right to form a topic row (see diagram below).
  4. Once complete, ask everyone to write 10 Things to Fix, this time on 10 blue post-its. These should be longer-term issues AND things to change for the next day. It is important to capture both.
  5. Repeat step 3 on a new bit of empty wall space under the title “What to Fix”.
  6. If you need, discuss and add a star (or do some dot-voting) next to key fixes for the next day/session/round. This will represent a clear group-defined priority for the design team.
  7. Finally, after both exercises, re-organise the rows in order of frequency and do a bit of theming and clustering to combine similar topics and tidy everything up (it will get messy as the post-it’s and rows grow).
Instructions for post-it organisation during the 10:10 workshop.

This is your top-line for the day. You can either take photos, or write them up easily to send round to everyone who was there (and those who couldn’t make it).

By running these mini-workshops at the end of each day, you will notice a few things happen:

  • Clear, obvious insights emerge from topics that were duplicated the most.
  • Clear, obvious fixes also emerge, along with some that are more left-field.
  • Most attendees will have a different take on similar ideas. This leads to important, positive and fruitful discussions.
  • By forcing 10, you move beyond the obvious, which is almost always a different place for each person.
  • You give equal space for every attendee to input, no matter the seniority (or confidence). Very often, the quiet ones will have the best, most considered ideas.
  • You are implicitly implying that it is not the researchers who uniquely own the rights to analysis, but the whole dispersed team. And with that comes a more rounded set of learning’s and solutions to take forward.
A 10:10 workshop in full flow. In this case there were so many new topics we had to start a new column.

So there you have it. If you bring all of these techniques together, you will have truly maximised the potential for insight and solutions. By curating the sessions into an engaging interactive event, you create powerful, insightful, and memorable experience for everyone involved. And most importantly you will have learned everything you can from both the participants and the observers in equal measure.

Thanks to so many people who have helped over the years. Our clients who pushed us to innovate, my bosses who gave us the freedom to experiment and my colleagues who weren’t afraid to push, pull and prod at existing processes. It is only by trying new things that we drive our field forward. I hope these techniques resonate and help in that respect.

As always, feedback, comments thoughts welcome.

The UX Collective donates US$1 for each article published in our platform. This story contributed to UX Para Minas Pretas (UX For Black Women), a Brazilian organization focused on promoting equity of Black women in the tech industry through initiatives of action, empowerment, and knowledge sharing. Silence against systemic racism is not an option. Build the design community you believe in.
UX Research
User Testing
Innovation
Design Thinking
UX
Recommended from ReadMedium