Following on from my earlier post about the different metrics applied to heatmaps I thought it was time to vent some frustration I have about a piece of work I saw posted on the internet. The example was a number of heatmaps showing performance of a number of print adverts. The images themselves showed hot spots in all the right places on the image – and the study said that the adverts were a great success and this was proved by the heatmap. We are always a little sceptical about the findings one can draw from a single heatmap so I did a little digging…. and when checking what the heatmap represented I found out it showed the number of fixations (as “this was the best measure for checking print / web performance” apparently) and therefore the images ‘worked’ as there was a lot of fixations in certain areas. As from our previous posts you may know that we scowl on the use of a single metric or output to try to draw conclusions (if not please read our blog post here : http://acuityets.wordpress.com/2009/10/03/heatmaps-the-truth-is-out-there/ ) so I thought a blog post would be appropriate  to show how utilising a single metric can prove to be misleading.

Firstly let’s look at a heatmap :

This heatmap shows the popular website for Woolworths, recently re-launched. This was part of a task to find a fancy dress costume for a child and this is the toys page, about halfway through the task for most users. While it wasn’t part of the task this heatmap shows a lot of interaction with the bicycle and special offer window, and a lot of interaction with the whole search bar along the left hand side. The mouse clicks show that there were several points on the left navigation bar (6 in fact) where people clicked through to try to find the costume – a point that maybe highlights that the labels / categories weren’t clear, as the task was quite clearly defined. What you may possibly say about this is that the special offer area drew attention (and therefore ‘worked’) and that people read and looked at each section of the left navigation. There was also a considerable amount of fixations on the top navigation bar in certain areas. This test was operated with the standard “Tobii Fixation Filter” which defines a fixation as gaze points within a 35 pixel area, with no minimum dwell time – what this basically means is that is the persons gaze remains within a 35 pixel area, for any period of time it is classed as a fixation. While this filter is a good ‘one size fits all’ filter you will end up with a lot of potentially very short fixations. Depending on which research you are basing your fixation metrics on (or by what measure you deem a fixation) fixations that show engagement or cognitive thought are deemed as ranging from 80ms - 200ms+. Therefore to rely on an output that could be showing a lot of very short fixations ( many potentially <80ms) is possibly ‘jumping the gun’ a little. What do I mean by this? Well very simply if there is a lot of very small fixations of 10, 20 or 30ms or so it is likely that the participant is scanning around the page but not actually taking in what they are seeing. After testing we would validate if they have actually seen, absorbed and remembered the images / page / advert by asking questions to test their recall and we will probably find out they didn’t – they were scanning looking for a keyword or link and didn’t see it, and were merely ‘passing through’ the area almost. Let’s have a look at a second heatmap from the same participants, on the same page, covering the same amount of time  only the type of heatmap metric has been altered :

You will notice immediately the difference in the outputs – and instantly if you were drawing conclusions or writing a report based on this output your findings would be very different. So what does this heatmap show? This is showing the amount of time actually spent by each person (relative to their exposure to the page - basically a % of the total amount of time on the webpage / stimuli). What we see instantly is the ‘hot spots’ are in very different places – and the 50% off area, the top navigation area and also the left navigation bar received much less attention than you would have initially thought if you had simply looked at the first heatmap. What ths actually proves is that in the initial image people were quickly scanning around th page, creating a number of fixations (and therefore hot spots) but weren’t actually absorbing the message. We now know that people didn’t actually spend time dwelling on the 50% off offers, and that people only dwelled on the options on the left menu that they clicked through on – proving our initial hasty conclusions were not correct. In this case the large number of small fixations on the left navigation bar were probably caused by the fact the menu is not in alphabetical order, and people were quickly scanning to firstly check how the menu is laid out and then secondly trying to find the link the wanted (or expected to see). All of this should obviously be backed up by our findings in our post recording questioning, gaze replays, and statistical data. Once again, no one report or output gives us all the answers! The overall picture of interaction is very different but this is a much more realistic measure than our original fixation based heatmap.

So to conclude – heatmaps are very powerful visual indicators of interaction and performance of a stimuli to some extent but only if used correctly. The type of fixation filter applied, and the actual fixations themselves are key to the output and this will radically alter the data presented. Just showing a fixation heatmap does not give any solid findings, it also doesn’t show accurately if and how much information a participant has absorbed – this needs to be examined post testing with probing questions, based on the key objectives and task at hand. Any thoughts or comments? As always contact me at jon@acuity-ets.com.