Ask DN: CTR dropdown with Medium style underline?

almost 4 years ago from , Design Lead at Graphiq.com

Hey DNers,

We recently ran an A/B test between the traditional CSS underline (text-decoration: underline) vs. the Medium style underline , which uses background gradient and looks way better.

We thought there would be no significant difference in the UX metrics. However, there's a 9% decrease in CTR with the Medium style underline.

Anyone else ran test on this? Thoughts on why that would be the case?

Thanks, Samantha


  • Mitch Malone, almost 4 years ago

    We need to know more about the test.

    • What exactly where you testing? All anchor tags on a webpage? All links in a product? Or specific CTA on a landing page? If you're measuring all anchor tags, it could be that had the users shown Medium style links navigated to pages that didn't require them to click on a link. This can skew the CTR and I'd probably rethink what you're trying to learn.
    • What was the total click volume for the experiment period? Small data sets produce extreme results.

    If you were experimenting on a specific CTA (where optimizing for CTR makes sense) with a solid click volume, I would pair these results with qualitative research, something as simple as just interviewing some people because 9% is pretty drastic.

    1 point
    • Samantha ZhangSamantha Zhang, almost 4 years ago

      It's the title in our data visualizations like the "Stephen Curry 2014-15 Playoff Shot Chart" in this example here: https://w.graphiq.com/w/315cxySqXzL

      Sample size was a few thousand, large enough to call it significant. We decided to move forward with Medium style for now, but that's definitely a strange test result to me still.

      0 points
  • Suganth SSuganth S, almost 4 years ago

    I think it's hard to say without seeing the content because there can be other factors involving in that. That being said, Safari has updated the underline styles similar to Medium in Yosemite release I think. So theoretically it shouldn't be a problem, never the less, do you feel the target audience might have felt the under line may be decoration and not hyperlink?

    0 points
    • Dylan SmithDylan Smith, almost 4 years ago

      Snap out of the bubble, homie. Plenty of people – even on Macs – don't use Safari.

      0 points
    • Samantha ZhangSamantha Zhang, almost 4 years ago (edited almost 4 years ago )

      Yup. See the example in my comment above, should be pretty clear that it's an underline not a decoration tho.

      Safari users aren't the majority of our user base.

      0 points
      • Suganth SSuganth S, almost 4 years ago

        I have a feeling that there is a fundamental problem in this. Black text is usually perceived as title here and I really doubt people hover on the text to find out whether it's clickable or not. I really feel you should try one test where there is an underline upfront; and then you can test out which one works out - medium style or non medium style (Note that medium style works because it's an improvement over the existing underline, visually; It doesn't fix the parent problem that you have here)

        0 points
        • Samantha ZhangSamantha Zhang, almost 4 years ago

          Yup you are definitely right. We were testing no underline upfront, traditional underline upfront, and medium style underline upfront. If we set the no underline version as the baseline, traditional underline brings 21% uplift in CTR, but medium style underline only brings 11%. And that caused my confusion.

          0 points