jump to navigation

Neuro-Tools: Emotion Detection January 16, 2017

Posted by eyetrackrob in Biometric, Captiv, neuromarketing, Tips And Tricks, Uncategorized.
Tags: , , , , , , , ,
1 comment so far

Much of the research that requires biofeedback involves emotions and the detection or classification of those. But as you surely noticed emotions are a very complex topic with many different aspects and challenges. Measuring when and how strong these events occur is relatively easy using electrodermal sensors, cardiovascular sensors or respiratory sensors and a software in which to process the raw data (and maybe even applies emotion detection algorithms – as TEA Captiv does). Things that I covered in my previous posts.

But immediately after knowing when and how strong an emotion occurred, the inevitable question comes up: “was it a positive or negative emotion?” (Valence) with the usual follow up question: “which emotion was it?” (Classification).

We have seen that each tool has its merits when it comes to finding emotions, but most biofeedback sensors on their own have the limitation that they can’t really give us answers regarding those rather interesting questions about valence or classification.

However countless studies exist, that cover specific emotions or sets of emotions and that use different sensors to measure the bodily reactions to those emotions. If we could just review all those studies, we could surely come up with a map showing which emotions could best be captured with which sensors and how the measurements might differ from one emotion to another so that we could identify specific emotions (i.e. fear, happiness, anger etc.) from those measurements.

emotion-detectionIdentifying emotions just by looking at the different biometric measurements. Is it possible?

Sylvia D. Kreibig probably had the same idea and reviewed 134 publications that report research of emotional effects so that you don’t have to. Her review “Autonomic nervous system activity in emotion” was published in 2010 in Biological Psychology holds some interesting findings as well as food for thought.

Before getting to the eye opening results, there are a few take-aways from her research that might be interesting.

  1. Generally most research is done on negative emotions. And in general negative emotions have been associated with stronger autonomic reactions. However in her review she did not report on magnitude of changes partly for reasons described in #3).
  2. Heart rate was the most used measurement in those 134 studies, followed by GSR and much more than HRV or Respiration.
  3. Baselining! Do it! Some of the studies she reviewed did it, others didn’t. There are a number of ways to do a baseline: neutral or benchmarking/variable. While there is no definitive way to do it (making it more complicated to compare between studies), the important thing is that you use some kind of baseline to compare your results to.
  4. A rose is a rose is a rose. But in emotions the same term describing an emotion can mean different things. Disgust can be induced due to contamination (dirty toilets, fowl food) or due to injury (mutilations, blood). Sadness can provoke crying or not and there are many other ambiguities: although anger for example is always a negative emotion it can drive people either away (withdrawal/avoidance motivation) or pull them closer in an aggressive move (approach motivation). Emotions that are semantically close such as fear and anxiety or amusement and happiness might still be based on distinct behavioural systems!
  5. Artifacts may influence the measured response. Watch out for the effects of posture and movements, ambient temperature and cognitive demands! Start with sensors that give you good signal quality to start with. If you then use TEA Captiv you can process the data, apply artifact detection algorithms and filters to smoothen the data and eliminate unwanted effects.

There are a few more things that need to be considered when comparing data from different studies but these are my personal top 5 take-aways. Apart from the results of course.

In the table below you can see already my summary of her results. In her review, she reports that HR was increased in most emotions and surprise. However in her review she comes to the conclusion that the HR decreased in emotions that involve an element of passivity such as non-crying sadness, contentment, visual anticipatory pleasure and suspense.

GSR was increased in most emotions probably reflecting motor preparation and increased action tendency and of the more commonly induced emotions in (commercial) research just non-crying sadness lowers GSR. All other emotions tend to increase the reaction.

HRV has been shown in quite a few studies to be useful as an indicator for cognitive workload. Low HRV usually correlate with means high stress levels. Since her review was mainly focused on emotions, cognitive workload was not considered and the use of HRV was not too helpful.

emotion-chart
The table shows different emotions and how they influence measurements, which can increase (+), decrease (-), depend on different factors (D) or are indecisive (I)

So, what does this table tell us? Is there a specific fingerprint of biometric activities unique to each emotion?
Maybe!
Under very controlled conditions and also taking into account other types of measurements there might be potential to discover a unique signature to some emotions.

Unfortunately for many researchers, very distinct emotions such as Anger and Happiness or Joy and Disgust have very similar bodily reactions – if we look only at HR, GSR and Respiration Rate. Different types of sadness can cause a variety of reactions, making it actually a very interesting research subject in my opinion, but this doesn’t make it easier for your everyday research and especially when it comes to commercial research you might not be able to control for every possible factor and to use too many sensors.

My personal conclusion is that while tools such as GSR, HR, Respiration or Heart Rate Variability can help us detect emotions, in most research projects they don’t allow to uncover which emotion it is and not even if it is a positive or negative emotion.
But on the positive side, we still have a few other tools in our Neuro-Toolbox that can help us along the way: Facial Expression Analysis for example, Implicit Association Tests or even EEG can help us to understand emotions, associations and motivations and thus help us to detect valence or even to classify emotions.

With this in mind, I’ll be covering Facial Expression Analysis in my next post as it is probably the easiest to use out of the three.

 

 

If you want to dig deeper into the original report, you can find it here: http://www.sciencedirect.com/science/article/pii/S0301051110000827
Sylvia D. Kreibig has since then been involved in some interesting research projects following up on the results. Take a look at her work on Research Gate.

Neuro-Tools : Heart Rate & Respiration November 21, 2016

Posted by eyetrackrob in Biometric, Captiv, eye tracking, Market Research, Marketing, neuromarketing, TEA, Technology, Uncategorized.
1 comment so far

Although not as fast as I thought, step by step, I’ll be covering the most relevant biofeedback sensors in this blog series. So far I’ve only managed to write about GSR, one of the sensors of the hour! Galvanic Skin Response has been around for a long time and in the past years it has gained lots of attention from researchers, but as you might have read in my last post, although it deserves all the attention it gets, it’s not always that simple to use.

Other measurements mentioned before that could tell you more about emotions or cognitive workload are respiration, heart rate and from this also the possibility to calculate the variability of the heart rate (HRV).

Heart Rate

Heart Rate (HR) reflects the frequency of a complete heartbeat within a specific time window. It is typically expressed as beats per minute (bpm). The HR is constantly, antagonistically influenced by the sympathetic nervous system (SNS) and parasympathetic nervous system (PsNS) and in general heart rate, similar to GSR, unfolds rather slowly. Although with peak effects observed after 4 seconds and return to baseline after about 20 seconds it is much slower than GSR. Heart Rate Variability (HRV) on the other hand expresses the quick variations of the frequency between heart beats. The time between beats is measured in milliseconds (ms) and is called an “R-R interval” or “inter-beat interval (IBI).”

ecg-signal

Image 1: shows a typical heart rhythm as recorded by an electrocardiogram (ECG). You can see heart rate (4bpm) as well as the differences in the inter-beat intervals.

Both measurements (HR and HRV) are closely related to emotional arousal, with HRV allowing for assessment of more sensitive and quicker changes, which also can be related to stress and cognitive workload (this might be a good topic for a follow up post).

While today many fitness devices exist that measure heart rate in the context of fitness and well being, those solutions might not be the ideal for your research. One of the reasons for this is the processing and averaging of data going on in the sensor.

fitness-monitor slide1

Image 2: shows the same recording as averaged data export (blue) and as it was displayed during the recording (orange). The data was recorded with a wrist worn device measuring the HR optically using light. In the averaged data the highest heart rate is at around 100 bpm. In the live stream the same time frame shows much more variability (still averaging at around 100 bpm) and it’s clearly visible that it is not the highest value of the recording.

 

As mentioned above, heart rate has a relatively low sensitivity and slow response. Many wearable fitness trackers don’t allow to export the data for further analysis or allow to access only averaged data, where quick spikes in the data have been eliminated as noise. The result of this prepossessing of data is that the effects of emotion might be lost altogether. On the other hand to compute HRV correctly, continuous and precise measurements must be guaranteed. Just 2-3 missed data points can mean inaccurate calculations of the times between beats and thus again missing relevant events.

slide21

Image 3: In the live visualization the highest heart rate reaches 145bpm. However the suspiciously round form reaching to the peak value indicates that data points are missing and data was interpolated. This becomes clear when looking at the averaged data. This data would not be suited for interpretation of HR or HRV.

Another reason why many heart rate trackers available for fitness purposes are not necessarily a suitable solution for researchers is that most of them are worn on the wrist and use light to measure blood flow and from there derive the heart rate. Compared to sensors that are placed close to the heart and measure electrical impulses (electrocardiogram/ECG), sensors on the wrist have to overcome challenges of compensating for movements, muscle-tensing, sweating and potentially light interference. ECG sensors are therefore the recommended tool for data collection for research purposes as they are more sensitive to certain signal characteristics.

ecg-beltecg-electrodes

Image 4: ECG Sensor as belt or as electrodes

Respiration

Research has associated respiration rate and depth with emotional impact and emotional valence. Interestingly olfactory information ascends directly to limbic areas and is not relayed through the thalamus as other sensory input. The Thalamus is a part of the brain which is acting as a relay and pre-processing for sensory information and is accounted to be relevant to regulate consciousness, arousal, wakefulness and alertness. As olfactory information is not relayed through this part of the brain, there is a different mechanism to make olfactory information conscious which leads to quicker physiological response and unconscious alternation of the respiratory pattern. Respiration patterns therefore allow to identify potentially unconscious liking or disliking and arousal. The deduction of a unique emotion from respiration rate and depth does not seem to be possible although more research is still needed in this area.

Respiration measurements can be obtained either from the use of dedicated clinical instruments, stretch sensitive respiration belts or can be calculated from ECG data. The latter being the least invasive for commercial research.

t-sens-respi-belt

Figure 6. Stretch Sensitive Respiration Belt

ECG data can be processed in TEA Captiv to obtain HR, HRV and even respiration rate and as with GSR all of the mentioned measurements can be synchronized with eyetracking to understand what visual information influenced a rise in HR, a change in HRV or an alteration of respiration patterns.

In my next post I’ll take a look at how all these measurements can be combined and if through a combination it is possible to not only detect emotional events but also understand whether it is a positive or negative emotion and even which specific emotion it is. So, watch this space for more!

 

Neuro-Tools : GSR October 24, 2016

Posted by eyetrackrob in Biometric, Captiv, eye tracking, Glasses, Market Research, neuromarketing, TEA, Tobii, Uncategorized.
3 comments

As mentioned in my first introduction to this blog, the central nervous system is divided into different branches which monitor and control different body functions. One of the branches, the sympathetic nervous system (SNS), is responsible for quick fight or flight reactions. By constantly accessing the surroundings and scanning for situations that could potentially be dangerous an evaluation takes place which leads to preparations for an adequate fight or flight reaction. These preparations can be measured throughout the body and include changing heart rate, respiration and levels of sweat on hands and feet.

As we start to understand that these non-conscious reactions are strongly and inseparably tied to decision making processes and thus human behaviour, more and more researchers have become interested in using tools to measure these reactions.

In my first post a few weeks ago, I wrote about the general rise of Neuro-Tools and mentioned some such as eyetracking, EEG, facial expression analysis, GSR, heartrate and respiration as well as Implicit Association Tests as examples. The series aims to go through these tools one by one and review what they measure, how they work and of course also where we run into the limitations of those tools. With the general objective to give you a perspective on how these tools can be made a valuable addition for your research, I’d like to continue the series looking at GSR today. Initially I thought of talking about GSR, heartrate and respiration in this post as they could easily be summarized as “biometrics” or “biofeedback measurements”, but it turned out to be a quite long post, so I’ll split them down into individual posts.

Enough of the introductions! Let’s dig into the exciting world of biometrics starting with:

Galvanic Skin Response

GSR isn’t simply around measuring sweat, there is an awful lot more to it than that so before offering some general advice on what to look out for when considering to use GSR, I would like to explain the basics around this tool.

Electrodermal Activity (EDA), Skin Conductance (SC) or Galvanic Skin Response (GSR) refer to the ability of the skin to conduct electricity due to changes in the activity of the sweat glands and thus the secretion of sweat. Those changes are closely related to psychological processes and can be triggered by emotional stimulation. Electricity can be conducted when an external, unnoticeable current of constant voltage is applied, and with more moisture on the skin, electrical resistance decreases and skin conductance increases at a measurable level, although sweat might not necessarily be visible through visual observation.

Skin conductance can be divided into tonic and phasic activity. The level of conductivity of the tonic activity is constantly changing within each individual respondent, depending on their hydration, skin dryness and autonomic regulation in response to environmental factors such as temperature for example. Phasic response in turn are short term peaks in GSR reflecting reactions of the SNS to emotionally arousing events, mostly independent of the tonic level. For most of the time, we will be looking at these reactions which occur in the eccrine sweat glands.

GSR data is measured in microsiemens (μS) and the relevant phasic reactions can be quantified and analysed in different ways. Apart from the number of peaks occurring within a certain period after stimulus onset, peak amplitude, the time to reach peak value and the recovery time can be used for analysis. GSR can be used to determine strength of arousal but can’t be used to determine the valence (like or dislike) of a reaction.

eda-example

Image 1 is an example of data including tonic and phasic activity.

 

The density of sweat glands varies across the body being highest on the head, the palms and fingers as well as on the sole of the feet. Most tools that measure the GSR are therefore build to be used on the fingers, where this reaction is strongest. However some instruments on the market allow for measuring the change in sweat levels on the wrist which often results in poorer data quality but might be necessary for some experiments where the hands are needed to interact with objects (i.e. holding mobile devices/products or typing).

 

eccrine-sweat-glands-distribution-2

 

Image 2 shows eccrine sweat gland concentration. Red areas indicate a high concentration of eccrine sweat glands (glands.cm−2) allowing to measure sympathetic arousal of low intensity and minimal duration. Green zones indicate a low concentration of relevant sweat glands able to measure only events of high intensity (for example on the wrist). (N. Taylor; C. Machado-Moreira, 2013

 

Depending on the manufacturer and kind of system used for the measurements, sensors can be adhesive electrode pads that are already filled with conductive gel in order to reduce preparation time and to avoid electrode movement. Conductive gel is not mandatory but can improve data quality and ensure a good and stable electrical connection. Many GSR device manufacturers that provide systems for the use on fingers and toes, provide Velcro straps to place the electrodes firmly. In any case excessive respiration, movements and talking should be avoided as these can cause noise in the data or variations in the signal that can be misinterpreted.

tsensgsr           e4-front_light

Image 3 shows a classic sensor (TEA T-Sens GSR) that can be placed on the fingertips adjustable with velcro straps next to an Empatica E4 wristband. 

 

As written in the introduction, reaction times and strength are highly individual and therefore distinct for each participant and they can vary between 400 milliseconds up to 5 seconds after presenting a stimulus. In a controlled lab environment a calibration procedure can help to understand individual differences in reactions but might not always be necessary. It is not advised to use GSR in areas where many low and high impact events can occur uncontrolled at any time and can be mixed with all kinds of artifacts, as it might be complex, if not impossible, to relate an emotional arousal peak to a specific event.
If free movement is a requirement (for example in shopper research) it is highly recommended to calibrate the GSR reaction time and strength for each participant and to complement the GSR measure with a synchronized video and sound feed -ideally even with eyetracking- to understand the source of the arousing events. The synchronization of several feeds can sometimes be a challenge but there are solutions that allow either for a live synchronization or a post-recording-synchronization.

 

tea-synch

Image 4 shows a synchronized recording of different sensors such as ECG, HR, HRV, Respiration and Cogntitive Workload with eyetracking (top right) and an additional video stream (bottom right). The synchronization can be done for example using the QR code that is visible on the screen (top left) marking a synchronization point in video and sensor feed.  

 

t-log

Image 5 shows a TEA T-Log, a small and mobile device that emits a short flash of light that can be picked up by a camera or in the video of the Tobii Glasses marking a visible event in the video and a sync point in the sensor recordings.

 

How GSR raw data, filtered data and emotion detection works all synchronized with eyetracking, can be seen in the following short video, recorded from TEA Captiv. I also imported data from a wrist-worn GSR device but the data was not usable, which is why I chose to minimize those curves in the software.  As you can see in Image 2 the concentration of eccrine sweat glands on the wrist is low which very often means having a very noisy signal or the absence of a signal. To improve the signal quality it is recommended to get a minimum level of tonic sweating, for example through some physical exercises. Although I did this (as you can indirectly and briefly see at the very beginning of the video), it wasn’t enough to make the measurement from the wrist usable. For these types of study (researching and improving the emotional and visual impact of TV commercials), I would usually recommend to use a remote eyetracker such as the Tobii X2-60 as well as sensors worn on the fingers (T-Sens GSR or similar), however I also wanted to show that it can easily be done with a mobile eyetracker if needed as shown below:

 

In comparison you can also watch a video of a similar test (same commercials) using a remote eyetracker as mentioned above. You’ll notice similarities in the general gaze data but also in the arousal detection, although you might also notice that each participant has a slightly different reaction time and  the emotional threshold has an influence on how many emotional moments each person is experiencing:

 

There is still a bit more to know about GSR and we at Acuity are do offer training on methodologies, technology and best practices for your research. To give you a headstart on some of the things to consider have a think about these 4 questions and then maybe give us a call:

  1. Where will the data collection happen? Do you need to be completely mobile, or will it be a controlled environment close to a computer? If you go mobile, can you carry a small device to record the data or does the GSR device itself needs to store the data?
  2. What type of sensor do you need? Is it a viable option to use sensors on the fingers, or will you need to use the hands to hold something or type for example?
  3. Do you know how to analyse the data? GSR raw data is rarely usable. Do you know how to remove the effects of tonic activity and artifacts and do you need a software that can do it for you and find the relevant events?
  4. Do you need to synchronize the data with other devices and do you want to accumulate data over several participants?

In the next post I’ll be covering heart rate and respiration to wrap up the more commonly used biofeedback tools before taking on EEG, facial expression analysis, Implicit association tests and others. Stay tuned!

 

Neuro-Tools : Essentials September 23, 2016

Posted by eyetrackrob in Biometric, Captiv, Market Research, neuromarketing, Shopper Research, TEA, Technology.
3 comments

In recent years eyetracking has become a standard measurement in many research fields and with the “neuro”-hype many companies and universities have started to add direct and / or indirect measurements of the central nervous system to their research toolbox aiming to add an additional dimension to help understand human behaviour and decision making.

Far from being a complete catalogue of all the options currently available this series of posts will concentrate on the more practical, and commonly used, tools for commercial research – things such as salience mapping, eyetracking, facial expression analysis, electroencephalography (EEG), implicit association tests and galvanic skin response (GSR).

With the dawn of wearable fitness devices that can easily measure blood volume pulse (BVP), from which heart rate and heart rate variability (HRV) may be derived, access to these measurements have become much easier, although not without limitations as it will become clearer in this series of blogs. Additionally some of those wearable fitness devices do allow some measurement of measure electro-dermal activity (EDA) and skin temperature showing that this technology is not far from mainstream use, at least in some form.

fitness-monitor

Although the word “neuro” is very often thought as a synonym for “brain”, neuroscience comprises the study of the complete nervous system and the tools and techniques involved are suited to measure directly or indirectly certain aspects of the processes occurring within. These tools can be broadly divided into three categories : neuro measurements, behavioural measurements and biofeedback measurements. The latter is as good as a starting point as any.

Our nervous system is quite complex and can be divided into different branches which monitor and control different body functions.

One of the branches, the sympathetic nervous system (SNS), is responsible for quick fight or flight reactions. By constantly accessing the surroundings and scanning for situations that could potentially be dangerous an evaluation takes place which leads to preparations for an adequate fight or flight reaction. These preparations can be measured throughout the body and include changes in heart rate, levels of sweat on hands and feet and respiration.

sns

The reactions of the SNS are not immediate to the exposure to the stimulus to be evaluated. Reaction times and strength are highly individual and distinct for different measures. They can vary between 400 milliseconds up to 5 seconds. As part of the fight or flight reactions, the change in sweat levels on the palms and fingertips is thought to be an evolutionary mechanism allowing a firmer grip. Interestingly this reaction can also be measured on the feet!

Changes in pulse are associated with changes in either physical exercise or arousal. If physical exercise is constant, heart rate variation can be a reliable index of arousal. Research has been conducted measuring different combinations of HRV and heart rate related to stress and to the identification of positive or negative valence and even specific emotions.

A third measured physiological measurements is respiration. The perception or anticipation of odours is depended on respiration. In other words our sense of smell and therefore emotional activation through it, is enhanced by respiration. Research has associated respiration rate and depth with emotional impact and emotional valence.

tsensgsr

At Acuity we provide tools to measure biofeedback synchronized with eyetracking to help understand not only where people are looking but also the emotional impact that it is causing. We can provide a series of sensors from different manufacturers that can be brought together into Captiv L700, a software from our friends over at TEA ergo (click here to see a video of the TEA Captiv Software integrating a variety of neuro-tools).
We are also happy to help you with training to explain how those sensors work, what they are measuring and get you started on the analysis and interpretation side of things.

My next post will focus on GSR but I will cover other biometrics, EEG, facial expression analysis and complements to eyetracking data in the following posts.

Stay tuned or feel free to get in touch via sales@acuity-ets.com to learn more about how to use neuro-tools in your research.