The results from usertesting.com were pretty impressive. The tester was particularly articulate, eagerly expressing his expectations and reactions to the site. It was pretty clear, also, that this was definitely an “expert” user. I usually don’t see people editing the URL string during a usability test (how about, never?). I received the video and a written report less than an hour after I submitted my task to the service.
One of the other positives was that I was able to encourage the user as a final task to explore the site to get his impressions of the general functionality. In traditional usability tests, you usually don’t have this flexibility in time to allow such free-flow exploration– every minute is precious because of the effort involved in getting the user to begin with. This service makes access to participants so much easier that it makes participants into a commodity (in a non-objectified way, of course).
Negatives? Well, there is the obvious that you can’t follow up with any questions or direct the test based on their actions. You are also limited to 15 minutes, which could limit some types of real transaction-based scenarios. Perhaps the biggest drawback is the limited factors by which you can recruit users (age, gender, income, computer expertise, country). Considering that I personally do a lot of work around business processes, I’d like to be able to recruit users based on their professional background - for quality participants, I’d even be able to pay a premium for it. However, I’d recommend the service if you have a general-purpose site or relatively simple workflow.
Hi Matt -
Thanks for sharing the experience on your blog! We appreciate your honest feedback on our service. Here are some follow-up comments:
- Obviously there are challenges to an unmoderated and remote testing service, such as not being able to give direction. In the future, we may offer additional premium services where clients will be able to ask users follow-up questions.
- The recommended time of 15 minutes is to keep the video file sizes manageable for both the users (who have to upload them to our servers) and the clients viewing them. If you have a test that requires more time, we can help you set up tests with multiple parts where the same user completes all parts. Please let us know if you want more information on how this works.
- We do plan to expand the demographic choices available to clients (likely will include education level, occupation, hobbies/interests, etc.). For now, you can indicate any other requests at the beginning of the “Tasks” field on the sign-up form (e.g. **Tester must be xxxxx**). When a client requests an additional demographic, the test is made available to users based on the five demographics we do have. The users are then to self-select based on whether or not they match the client’s request. In the event that we do not users that fit the requirement, we will contact you with the option to edit your demographics or cancel the test for a refund.
If you or any of your students have any questions, please feel free to contact me!
Aimee Williamson
Director of Customer Support
Web: http://www.usertesting.com
Twitter: http://www.twitter.com/usertesting
Email: Support@UserTesting.com
Phone: 800.903.9493