This is interesting. In my (designer's) mind, the new design is WAAAY superior for a variety of reasons, so I thought that the user testing would prove this as well...
So would the perceived difficulty be higher because it's different than the standard airline booking flow? Non-traditional? One user cites familiarity with other booking forms.
How long would it take for a user base to be familiar with a non-standard (but possibly better flow)?
Or how would something like this stack for someone who has never booked an airplane before??
Just thinking out loud.
Your comment is pretty much on point with what I was thinking. The traditional airline booking site is a cluttered mess, but users are familiar with it now and can navigate through it. At what point do you start from a clean slate and prove that there is a better solution out there if you're willing to let go of old habits.
Same paradigm as Apple coming in and shaking up the mobile phone world. Everyone was used to the clumsy stylus or flip phone navigation, yet Apple figured out a way to bring something completely new, but still familiar and natural.
I personally love the redesign both aesthetically and functionally.
the new site is measurably slower, harder to use, and less enjoyable, but users prefer it anyway. ok sure.
Instead of assuming that means it's a bad survey, maybe question our own assumptions? Maybe speed and ease-of-use aren't the only factors valued. The top words on the new site were "modern," "fun," "clean," "friendly"...
true, but the data even said they enjoyed the new less on desktop, slightly more on tablet and mobile. (which is pretty bad, considering the old wasn't responsive.) yet when asked about specific comparisons of the ui, they overwhelmingly prefer the new.
more likely, they were too polite to say they preferred the old site.
People frequently prefer things that are slower or harder to use when the thing satisfies some greater need they have, like comfort, familiarity or delight.
To my surprise, my girlfriend recently told me that she prefers Windows 8 to Mac OSX. When I asked why, she said "Because it says hello to me when I sign in". She also prefers MS Word to Google Docs because while she can appreciate that Docs looks cleaner, she trusts herself with word and is confident that she can get her work done, even if it takes longer.
Most people are emotional creatures, not logical.
"All users viewed the old site then new site."
Anyone else thrown by this? Why didn't they mix it up some? Like half the subjects used the new site first?
Could the fact that they already figured out how to use the old booking system and then had to learn a whole new one be throwing off the results?
That's an excellent point. Would love to see how people react going from new to old.
It would be interesting to see the results of the new site with a user group that hasn't been biased with familiarity of the old one.
I wish the sample size would've been bigger.
Why? You already gained lots of insight about your users by just testing a few of them. I certainly don't agree with the "5 users will discover 85% of your usability problems", but even this "small" sample size (pretty big in my eyes) can provide valuable results. You have to be aware of qualitative and quantitative testing. This sample size will reveal lots of usability problems – better than 5 users would have done – but they are still doing qualitative research.
It's interesting that people found the old site easier but said they preferred the new. Generally, I feel that preferential stuff should be taken with a grain of salt in this kind of testing as someone could think "It's new so I should like it more than the old."
That said, it will be interesting to see what type of evolution the new design takes as they have more data on the usability of the experience.
Interesting research. I am still a bit cautious with the conclusions.
The problem with these kind of researches are, in my opinion, that the user really needs to think about each step/click/interaction. Because that is the task that he/she got from the researcher. A normal user doesn't think (and doesn't wants to think) about each step. This has huge effect on the results. That is way I like to trust and prefer my web analytics and a/b testing above this kind of research.
“If you had to book a flight to Seattle right now, which site would you book it on?”
Asking people what they would do.. People can't predict their future behavior. What they're stuck with are opinions.