Four people from around the globe were transported to the Dutch capital Amsterdam and put together as a team at the Amsterdam MediaLab to face the challenge from Triple. Yujie Shan (CN), Janina Saarnio (FI), Christiaan van Leeuwen (NL) and Agnetha Mortensen (NO) with guidance from their coach Tamara Pinos from Ecuador. All with different background and skills, in the prospect of learning from each other and growing together as a team!
GOAL (CHALLENGE):
“The development (and open-source release) of software that can determine the user’s emotional state based on sensory data (e.g., head motion, pupil dilation, heart rate variability) that is produced while she or he interacts with a virtual reality (VR) environment. This software can serve as a ‘middle layer’ between app and user, allowing the VR environment to the user’s present emotional state (i.e., valance and arousal levels).”
The last few weeks we have been attacking our challenge from different angles and trying to figure out different ways to meet our goals. In the startup week, we had a 3-day sprint where we got to play with the different tools in the Makers Lab and create different prototypes to show our client Triple. We’ve dugged deep into emotions, sensory technology, VR and Augmented Reality (AR) to try to understand our problem, get better insights into tools available and get an overview of our options and potential sololutions. Last week we went to Triple’s headquearter to meet our clients and talk to them about our ideas and hear their thoughts and inputs. What were they actually expecting out of us, and how could we collaborate together most optimally to achieve our goals? Triple is giving us a lot of freedom in how to tackle the challenge but will provide us with tools, and expert knowledge in order to reach our goals.
Getting an overview of our challenge and generating ideas for prototyping in the MakersLab
One of our prototypes made in the MakersLab. A form of Twister based on emotions, where the different emotions would have different textures that would represent the feeling generated.
After our meeting with Triple we decided to focus on one specific emotion that we can test and conduct experiments on. After an overall research on emotions we decided to focus on Fear, as this is an emotion that there is already conducted a lot of research on, and especially in relation to detecting that emotion through sensory technology. Everybody can get scared and feel the emotion of fear when something external triggers or fight-flight responses in the Amygdala in the brain. When you are afraid your heartbeat rises, you start sweating and body temperature drops by a few degrees. All this is trackable through different sensors, and our challenge now is how to select the sensors that best can detect the emotion of fear.
Janina and Christiaan playing around with skin-conductivity sensors
Today we had an expert consultation with Joanekke Weerdmester (PhD-candidate at the Developmental Psychopathology research group at the Behavioural Science Institute in Nijmegen), to share our progress, ideas and get insights into how to tackle our problem further. Joanekke’s PhD-project is focused on the development and validation of biofeedback games for children with anxiety, and we found her field of knowledge highly relevant for our project. She was very helpful in advising us on the different routes we can take with our project, and she gave us a lot of relevant information in relation to games, and sensory technology. She has previous experience with different sensors used in projects she has been working on such as the games DEEP, and Nevermind. Her inputs definitely gave us some ideas on how to progress further.
1st week prototype in how to detect emotions by self-reporting 4 basic emotions; sad, happy, scared and neutral.
The players would look at pictures in different categories where we selected 4 pictures that we thought would represent the different emotions. We would then see if the participants would report the same emotions as our assumptions. The prototype was made using MakeyMakey.
Janina and Rory on our way to our first meeting with our clients
On our way back to the MediaLab in Amsterdam after our first client meeting with Triple in Alkmaar. Everyone super happy while Christiaan and Agnetha are lost in the virtual world of distractions.
We are excited to continue this journey further, trying and failing, and hopefully, we can contribute to an amazing open source software that can benefit different stakeholders and research institutes 🙂
Until next time,
Meraqi,
Janina, Christiaan, Rory and Agnetha
(Team Sense VR)