Online Usability Testing Reviewed: Feedback Army & UserTesting.com

I finally decided to finally give a couple online usability testing tools a shot.  I figure it can't hurt, and would be fantastic if they actually helped!  For this review I started with Feedback Army and UserTesting.com.

Pricing starts at 10 answers for $15

My Plan:  25 answers for $30

Feedback Army works simply like this:  You submit a URL & 5 questions, and voila... you get immediate answers from real people.  Seriously it's that simple.  

I found it to be excellent for quickie instant reactions to your site, however the quality of the answers was not very good, nor did it have a completeness of thinking. It's really a version of the 5-second test, but actually that certainly has it's value.  The best part about this service is that the results are immediate, and you can start to see patterns emerge quickly.  That said, there is no filtering by demographic at all, so it's sort of a mixed bag of responses.  

The sign-up commitment is extremely simple and easy, and pricing is really affordable.  Perfect for an overview snapshot instant reaction to your questions. What it's not good at is the deep-dive into specific questions.  So good perhaps to find out what areas to focus on, or what questions to ask for a more in-depth testing service like UserTesting.com, below.

I do love how simple it is to get started, and they do a great job of providing helpful examples for your questions. You don't need to have any experience running usability tests to use this service. It's a nice, lightweight tool that I will probably use again.  But as I said I think it has a specific purpose -- and that is not a deep dive into specifics.

Results are provided in the form of a simple, long scrollable page, where answers appear in the order they are submitted.  The option to subscribe to the page as an RSS feed is handy.

Pros:

• Very affordable, as cheap as $15 (so cheap you can't help but to give it a shot!)
• Really easy to get started & really simple to use
• Hand holding approach and provided plenty of example questions for you
• Simple concept, don't need a huge amount of planning by any means
• Very quick results - first answers within the hour and they kept coming in regularly
• PayPal payment, which made my set-up way easier than any of the other services

Cons:

• Very general reaction type answers, nothing too thoughtful
• The answers and short & sweet, sometimes not even a full sentence. You need to be able to analyze this for patterns, and sort out what is valuable and what isn't.
• Some users provided feedback with improper grammar, which tells me that many users are offshore. In most cases this could be totally fine, but for my purposes our target audience is US based only, so that was something to consider in analyzing the results.  
• Because of the above bullets, make sure you ask the right questions and word it properly, to get the most valuable answers.

Pricing starts at 1 user for $39

My Plan:  3 users for $117

After the swift start from Feedback Army, I was hoping to have another quick setup, and I did.  UserTesting.com has a fantastic 1-page sign up which I love. It includes everything to get going, very refreshing.  First you add your test information, which they provide excellent examples to help you along, it's very helpful.  Then select your plan, enter payment info, and you're done. The only thing I missed was no PayPal option.   I paid for 3 users, and anxiously awaited the feedback.  Another nice option here is that you can specify the demographic of your target users, which Feedback Army did not have the option of doing.  Was easy to get started, and the demographic filtering plus video feedback to me, made it worth the much higher price.

One notes on the demographic filters, I initially was pretty specific, and only received 1 user response.  Waited 24 hours, and realized that was the problem, so I edited my test to make the demographic slightly broader, and received the other 2 responses within the hour.  The users were the right demographic and provided well thought out, detailed, specific responses. 

UserTesting.com is kind of the opposite of Feedback Army, in that you pay for fewer users to provide feedback, but the feedback is a total deep dive into your tasks.  The feedback comes in the form of a video with users' voiceover, talking through their actions & thoughts as they complete your tasks.  This was extremely valuable and fascinating.   Users are US based which I appreciate, and from what I can gather, had the demographic info I requested.

Pros:

• Fairly affordable, multiple price points
• Quick to get started
• Great 1 page sign-up for everything (project test details, new account, payment). Love it.
• Quality of feedback is excellent. Pretty good for blind selection online. 
• If users match your demographic info, feedback was within 1 hour, great!
• Ability to filter your demographic (but see con below)

Cons:

• No Paypal integration. Didn't know this would be a con until I had to actually enter in all my payment & billing info. Was far easier with Feedback Army to send them my money fast.
• Price is higher, and had to think harder if it would be worth it, ultimately the video sold me on the higher price tag.
• If your demographic filtering is too specific, you won't get many users.  However it's easy to update that info & resubmit.

Overall, although UserTesting.com is significantly higher in cost, I think you get much higher quality, relevant responses.  I would rather take 3 high quality responses than 100 low quality responses.  However both Feedback Army & UserTesting.com have a user in my opinion, and  I would actually use them as compliments to one another.   I think UserTesting.com is a great way to get valuable feedback without having to set up a full lab yourself.  Well worth the money in my opinion.