I, personally, find it interesting that this study doesn't work on mobile devices as a response to my article on mobile device icons.
Also adding elements like color and filled-background-hollow icons without a doubt biases the candidates.
The original article I wrote that started all of this follow-up research simply states that the icons take more time to interpret and that (over the course of the 3/4 different eye tracking studies I've seen pop up) has never been disproved. It's only labeled as insignificant because the numbers aren't massive (or don't agree with the author's opinions). Doesn't change that it's true.
I think the evidence presented supports his position much more than your criticism of this research. Do you have any evidence that a mobile vs desktop context would make a difference, and links to the eye tracking studies ? Your initial post uses research in how the brain processes words as a key pillar in your argument. I fail to see how this is contingent on a mobile context.
While I haven't read the entire paper to look at the methodology, at first glance it seems like it was quite rigorous. Eye tracking studies on the other hand are of questionable reliability if I'm not mistaken.
I think it's also worth noting that significance is a statistically defined term; not one that's arbitrarily defined by the author.
I disagree. Rigor vs accuracy and a qualified test are two different things.
"There was no noticeable difference in speed of recognition" 64.2 seconds vs 72.6 seconds.
8 seconds is not significant? That's like 10%. Again, this all still supports the original argument that they take longer. It's not debatable with so much research continuously stating they take longer which was the whole point.
None of these tests are accurately executed anyway. In the one above she read names of items and had users point to the shape that represented the icon. In the test from the OPs article it's done via the web only AND some weird add-on stuff like colors and different styles.
The tests are fine and all, but useless comparing them to the original assertion that looking at hollow / thin line icons on mobile devices takes longer (which it does).
I encourage you to read the full paper to fully understand the author's reasoning. He addresses your criticism about the different styles. For what it's worth he randomized the order of the icons which accounts for your claim of a influencing the results.
I don't want to make this a discussion of semantics, but you should know I meant scientific rigor.
based on your further comments I'm not sure if you understand statistical significance and this is making up a decent chunk of this discussion.
I'll also ask again, why does the mobile context matter for your claim?
I'm not a fan of hollow icons, but that's strictly my opinion. I challenge you to demonstrate that your claim that hollow icons or thin lines take longer (to what?) is true. You've made a claim with questionable scientific bases that can be empirically/scientifically verified. Unlike some other informal tests, the study posted is a very valid and imo an accurate way to test your claim.
I further challenge you to present evidence that supports your claim. Not just "because it does." The numbers that you presented (64.2 vs 72.6) are not appropriate comparisons. For that study, It's the delta between line vs filled for one participant group that you want to compare. If we go by the numbers you chose, line icons were actually faster for a trial (60.6 vs 71.4).
Edited for clarity. Missed a few words.
Worth check out Apple's official advice at (for the devices):
Video title: "Designing Intuitive User Experiences" - 45 mins in.
Apple's advice on the "down"state of the button is to use Solid icon -- in many case, it looks less good, but it helps on folks with color blindness or other accessibility issues.
Here is the actual research paper: https://dl.dropboxusercontent.com/u/5072835/Filled-in-vs-Outline-Icons-Curt-Arledge-2014.pdf
And here is the website used for testing: http://www.icon-test.net
So this is an interesting study, but I wish they would have taken it a little bit further. All they tested was recognition speed, which is not actually 1:1 with cognitive load. You can actually measure cognitive load directly though using Electroencephalography (EEG). An EEG headset like this one comes with software that returns cognitive load directly.