TL;DR I created a nifty system that automatically recruits and schedules users for regular user research. It’s absolutely swell.
User Research…we love to talk about how important it is, but when the rubber meets the road, it seems to be the part of the design process that most frequently gets short changed (if not completely removed.) I have spoken to a lot of people about their research process. It seems most of us exaggerate how much we actually do it. Ok, Maybe I’m not talking about you, but it’s true of nearly all the rest of us.
When I was at my last gig, I recognized that we needed to do more research. I got up on my soap box and shouted for all to hear. It was generally the product manager’s responsibility to set up research activities with users. At times, when my PM was too busy, I took the initiative to make sure it happened. Other times, I let it fall through the cracks because I was busy too, or I didn’t know how to find the right users.
I went through these ebbs and flows of doing research. I knew it was important and something that needed to be done. But it was hard. I was comfortable with the methods of collecting data, but finding the people to test and doing all the leg-work to set that up took a lot of time. Unfortunately, the UX team had gained the confidence of developers and the rest of the product team to a degree that they didn’t push us to do user research. Let just say, we got away with more than we should have.
From time to time the UX team would sit around dreaming about doing more research, but ultimately it wouldn’t happen because it took too much to make it happen.
When I interviewed with DirectScale, I was impressed with their desire to have a truly research driven product design process. I joined the company because we had a shared vision of how product design should be done. I was also excited with my prospects. They knew what they wanted, but they didn’t know how to get it. My role would be to turn their vision into reality. But if we were going to be research driven, we had to make doing research a priority. My first thought was to set a goal to talk to X number of users per week.
With a goal like that we would constantly be focused on doing research, great right? Nope. It sounds good on the surface, but I realized I didn’t want the team to be constantly focused on doing research. I want them to be focused on our users and the problems we are trying to solve. I wanted the research to be so ingrained in the process that it isn’t something that we have to force ourselves to do. It something that happens naturally as a way to understand users and solve problems. So I set a goal that might seem counter intuitive.
Don’t spend more than 2 hours per day talking to users.
This goal comes with the assumption that we are talking to users very frequently. Rather than tackling huge research initiatives on an irregular basis, and getting burnt out on marathon usability sessions, we are consistently doing small amounts of research. We always have our finger on the pulse of users; we are talking to them every day.
Now the hard question, how could I institutionalize this. In my last job, the belief that it was important was ingrained, but that didn’t drive the behavior. Getting everyone on board with the idea of testing frequently is easy, actually prompting people to do it (including myself) is where I struggled in the past. I needed to focus my efforts in reducing friction. My hypothesis was that if it was easy and straightforward, we would test frequently without external influences forcing us to do so. Lucky for me, I happen to spend most of my time figuring out how to make things easy and straightforward.
Step 1- Create a Process
It seems a lot of people are paralyzed by not knowing where to start. I started by figuring out the best way to identify and get in touch with our target users. When I started my job they already had a list from our clients of about 10 end-users that we could talk to. That was a start, but we had to figure out how to get more.
Almost every time I’ve done a user interview or usability study, people comment about how much they enjoyed it and express gratitude for being able to be part of the process. So each time we talked to someone, we explained how important it was for us to talk to as many different people as possible. Then, after the interview, we sent a thank you email with an invitation to invite their friends to participate in this community. We provided a link for them to pass on. This approach was helpful, but it didn’t give us the volume that we needed.
Step 2- Create a Panel
Our goal was to be talking to users daily and we know that we couldn’t be asking our clients for more referrals every couple of weeks. Instead we needed to create a panel of users that we could continually draw from. Talking to users daily would mean that this panel would need to be always growing.
We thought about some fancy ways that we could accomplish this by fully integrating it into our other systems. But we also recognized it would take much longer and that this was new territory for us. We wanted to prove the idea out first.
Instead of taking hours building an awesome integration, we decided to keep it simple.
We added a persistent tab at the bottom of our website that said “help us improve.” Clicking on the tab lead to a very basic Google form, essentially only capturing name, email and phone number. Those items were dropped into a spreadsheet and that was our panel of users. We actually had to create a different form for each of our clients so we knew where the users we coming from. We ended up having about 10 different sheets with all the users. Whenever we talked to someone, we would would make a note right there in the spreadsheet. It wasn’t great, but it was so much better than what we had.
Step 3- Iterate
After our first pass at getting this going, we were seeing some success, but there were several areas that we wanted to improve. We were happy with that, if our goal was to make it perfect right out of the gate it would have taken much longer to get it up and running and chances are it still wouldn’t be perfect. This approach allowed us to focus on changing the areas that would make the most impact.
Even though people were joining our panel, they weren’t doing it at the speed we wanted. Only a of couple people per week were actually clicking on that tab. We hypothesized the problem was either they weren’t seeing the tab or had no reason to click on it.
We decided to run a test. On just one of our clients, we would display a popup at login inviting them to join the group. We didn’t want to continually bug users and we wanted to keep the story small so we didn’t want to have to track who the pop-up should be displayed to, so we decided regardless of if you clicked join or just closed the popup, you would only see that pop-up once.
This is another example of doing things fast instead of doing things nice.
It would be nice to be able to identify in our system what users were enrolled in our panel, but that would have taken longer to implement and since we were relying on another team to add the code to the website, we needed to be as small of a story as possible. In the month that we had the tab displayed on the screen, about 15 people had joined our panel. Once we launch the pop-up window, we got over 200 in just two days. Obviously, we decided to implement this change across our other clients as well.
It didn’t take long for us to learn that managing all our panel and contact history in a different spreadsheet for each client was not ideal. We needed a single place manage all our clients that our whole team had access to. One member suggested using CRM. We didn’t need anything too fancy and found a free CRM that met our needs. Every few days we would go into our Google sheets, copy all the new people into a separate sheet, then upload that sheet to the CRM. Now we could track all communication and each team member would have access to it.
The last major thing that we needed to improve was scheduling. When we started it seemed to take forever. First, we would have to send out an exploratory email to find out if they were willing to meet with us. Once they replied we had to send a few emails back and forth finding a time that would work for both our schedules. Sometimes people got confused by timezones and ended up expecting is to call them at a different time. After all this, about half the time we couldn’t get a hold of them at the scheduled time and it was all for naught.
We were able to find a tool that improved this process dramatically. Calendly integrates with Google calendar. In order to schedule a time with someone, just send them your Calendly link. It will allow them to find a time that works for both of you.
Step 4 — Automate
We are a small startup. We could not afford a full time researcher to oversee the recruiting and scheduling of research participants. Regardless, we had to find a way to talk to users on a daily basis. Calendly worked great for facilitating the scheduling of a specific time, but what if we could automate all of it? That would be a game changer.
Our small product team decided to make this a reality. We set aside one day for a hack-a-thon. By day’s end, we had built a prototype for how we could achieve this, and created some plans to make it happen. Over the next couple weeks, whenever we could spare a moment between projects, we worked on getting this automated.
We outlined a flow of what we wanted to accomplish and used Zapier to connect all the tools we were already using. This approach was optimal because it was something that we could do without relying on any other teams to help us achieve our goal. Success was in our hands.
Upon completion, our automated system removed virtually all of the friction of scheduling users for research. Users are automatically added into our CMS, a persona survey is sent out to learn more about them and results feed right back into the CMS. When we want to schedule someone, all we have to do is go into the CMS and target users based on all the information we already have about them. When we are ready to invite them, we just select an option in the CMS. An email is automatically sent, Calendly automatically schedules the user and adds it to the team’s calendars. The team is notified via slack the we have a new appointment. All we have to do is show up.
This is still new to us, we just started using this new automated system last week. In our first trial run, we identified 6 people we wanted to talk to. It took about 2 minutes of our time. Of those 6 people, 5 of them signed up for times to talk to us, and none of them stood us up. We still have some tweaking to do and will continue to iterated. But I feel we are in a position to succeed in our goal of not spending more that 2 hours per day talking to users.
If you’re interested you can take a look at our full automation flows.