Heatmaps – The Truth Is Out There! October 3, 2009Posted by Jon Ward in Uncategorized.
At Acuity we constantly strive to improve our customers knowledge on the functionality of Tobii Studio and the Tobii eye tracking hardware, and today we are going to look at heatmaps – probably the most commonly seen visual output from eye trackers in the world! A lot is said about heat maps and what they represent, in fact some consultancies may use nothing but heat maps in their reports, but surely there is more to eye tracking than a few pretty pictures??? Of course there is! Heat maps, like every other tool in the eye tracking arsenal are only part of the picture, heat maps are used to identify areas where there is a large amount of attention, either time based or in terms of numbers of fixations. They don’t say “this worked” or “this didn’t” you need to combine this output with other methods and metrics. For print work for example – the heat map may show a lot of attention on the new “XYZ” company logo – so it works right? Well no, because using the AOI tool and statistics in Tobii Studio we see that although everyone fixated on the logo, the average was 0.100 seconds, and when we questioned the participants afterwards to check their recall only 10% of the participants remembered “XYZ” even existed – but the heat map was all pretty and red wasn’t it? Oh dear what went wrong???
The answer is – nothing! As we trawl the internet looking for eye tracking blogs, reports and pages we see scores of heatmaps put up to prove a point – but very rarely are they put into context… how many people realise there are three different types of metric to apply to heatmaps? And that depending on how you have defined a fixation in the filters setting this will also affect the results? And what exactly does the red bit on my homepage mean?
As part one in an ongoing (irregular) series – let’s look at three different heatmaps to illustrate one point…
So here we have what looks like three different outputs showing different behaviour… or do we? The answer is no, each of these heatmaps represents the same 3 participants, over the same amount of time, on the same page. So why are the outputs so different? Simple, each one is created using a different metric within Tobii Studio and without illustrating which metric we are using – the reader is no wiser…..
The first is created using the ABSOLUTE DURATION setting, this works by adding the total amount of time spent FIXATING by all the participants you currently have selected (in this case the ‘hot spot’ is 10.44 seconds) – which shows the actual amount of interaction your test participants had with that area of the stimuli. Which is very useful – the longer the spend the more likely they are to recall or the more engaged they are…. well maybe – consider the fact that they may have spent a lot of time looking there because the call to action wasn’t clear, they didn’t know where to go next, or the image / text was just plain confusing – just because they spend a lot of time looking there does not make it good interaction – it is purely interaction that needs investigating. Also the problem with this output is that if you have (for usability testing in particular) a person who spends 5x the average period to complete the task / read the page their data will skew the other results potentially. However for stimuli displayed for a set period of time this metric is very effective.
Our second picture is created using the FIXATION setting on our heat map button. This is the default in the software and the one we assume most people use (although it is usually very hard to tell as hardly anyone labels their outputs to put them in context!) and details the number of fixations made in each area by all the selected participants. Again we have to consider that a lot of fixations may be a good or a bad thing. We also need to consider where people entered the webpage or stimuli, as the first fixations per person are probably a legacy from the previous page – and of course these fixations will offset the other data on the page. Consider looking at removing the first 1/4 – 1/2 second of exposure to the page on your timeline and see if the outputs varies. Alternatively try to control where people come onto the page – a central fixation marker or similar is useful for print – and then you can comfortably say that the first ‘x’ seconds of your visualisation can be discarded as this is the transitional period between images. Use a gaze plot to see where people came onto the page and use this to window your outputs correctly. Also look for anyone who has an eye condition that may cause rapid eye movement, like nystagmus for example – where we have witnessed 60 – 100 fixations per second!, as this can also offset data.
The third and final setting is done using the RELATIVE DURATION filter and is ideal for web pages. This great tool normalise each individuals data across their journey on the stimuli. Primarily used for web testing this removes the issue detailed in the ABSOLUTE DURATION section above by looking at each persons exposure to the stimuli and instead of using time as a measure it uses a percentage measure. So if you were on a page for 2 minutes, and spent 1 minute looking in a certain area that would be 50% of your relative time on that page. By applying this to each person individually and then aggregating the data you get a much more accurate output for web tests. It removes the problem of people seeing images for different periods of time and gives you a better feel for peoples interaction regardless of their speed through the testing.
So to summarise this post (and part one in our ongoing series about heatmaps!) – there are a lot more to heatmaps than most people think. And contrary to what some people may say – heatmaps do not give you the answer to life, the universe and everything (the answer to that is 42) but they do provide a great graphical output to show your clients to say ‘…..you will notice the red area here…. unfortunately that does not illustrate the link worked, it actually shows how people spent a lot of time in that region because they didn’t know where to go to next. We used this gaze plot and the statistical data we generated to show that 60% of participants looked to the right for the next call to action. Our recommendations – backed up by this data – are to change the layout…..’ – you get the idea! The red hot spots on a heatmap give you a starting point, a clue as to where to start looking and a talking point with your client – but a good practicioner will have the ability and knowledge to use these to illustrate their thoughts and conclusions based on a wide range of other outputs, observations and methodologies.
Eye tracking is a hugely powerful tool, and the software gives you a whole range of weapons to attack the data with – but the outputs and data need to be used together to create a full and true account of what happened, just printing a heatmap is not enough… dig deeper and you will get much more from your Tobii.
As always – feedback is appreciated, there is also a Linkedin discussion about heatmaps here : http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&gid=1048797&discussionID=6948064&goback=%2Eanh_1048797 which you may be interested in. If you have had your Tobii system for a while and want a skills refresher, a demonstraiton of the latest features of Studio or you have realised you need to buy a Tobii (or two!) then contact me directly via email@example.com.