UNC computer science students invent way to control a video game with brain waves
Most people you pass on the street have a general understanding of augmented reality/virtual reality (AR/VR). But a team of undergrads at the University of North Carolina at Chapel Hill, led by Brian Moynihan of the UNC Health Sciences Library, are not only creating a new way of thinking about these technologies, but exploring new possibilities for thinking with them.
The students and Moynihan developed the Brain VR project, which combines electroencephalogram (EEG) data with AR/VR and allows users to control a video game using only their minds. Users train the system by thinking “up” or “down,” enabling them to fly across a virtual landscape using only their thoughts. While many people are familiar with the capabilities of AR/VR, Moynihan wanted to bring this technology into the classroom to challenge students to explore different possibilities for how the technology could be used.
“I’m interested in how we use technology to improve health, particularly with wearable devices that incorporate biofeedback,” says Moynihan, Head of Health Technology and Informatics at the UNC Health Sciences Library. “One tool is the EEG monitor, which uses four points on the head to capture brain wave data. I wanted the students to figure out how to control the game using just two states, thinking up and thinking down.”
The Brain VR student team included Carolina undergrads Jonathan Monroe, Mark Molinaro, Ronald Ding, and Grady Hale.
“The students had to do a lot of groundbreaking work over the semester to make the Brain VR project come together,” says Moynihan. “From pulling and figuring out the data from the EEG headbands, to translating that data into code to make it work in a virtual reality environment.”
As the students worked to translate and apply data collected from the headbands, they faced a daunting task. EEG data can be very “noisy” to work with, especially when it comes to VR. Many times, more noise than signal is produced. In a laboratory setting, users can sit, relax and read while EEG data is pulled. In VR, users are getting up and moving around, which can affect the data.
“The student team had to be innovative in how they extracted the data that was needed,” says Moynihan. “So, they wrote code to help with finding that specific data, filter out the noise, and then creating the outputs that were needed into VR.”
The Brain VR project even allows users to relax by a campfire under a starry night and keep the campfire under control using their brain waves. It also offers a 3D representation of a brain in VR where users can walk around the brain to see live brainwaves being emitted based on their EEG data
“Innovation is not just purely academic,” says Moynihan. “Not only did the students write the code to drive this novel technology, they made their work open source so that others can use it to build upon, tweaking it as needed for their individual projects. In fact, the code has already been used by other Carolina students to create serious games.”
As the Brain VR project came together, Moynihan and Monroe wanted to share the project with the wider public as an example of innovation taking place at UNC. With support from Innovate Carolina, Brain VR was able to participate in the Triangle Sci-Tech Expo at the North Carolina Museum of Natural Sciences in Raleigh, which attracted more than 5,000 visitors from across the region.
“What is notable about that event is that people from all walks of life attended – from computer scientists in Research Triangle Park to elementary school kids – everyone was excited to experience VR,” says Moynihan.
Arriving early and staying well past the end of the event, Moynihan and Monroe were peppered with questions from curious visitors. The event gave them the opportunity to share their excitement about AR/VR as well as expose people of all ages to the cutting-edge research and innovation happening at Carolina.
To collect and capture brain wave data needed throughout the project, the students worked with the Muse brain sensing headband. The headband is mostly used for meditation. However, Moynihan realized data captured could be used for research as well. Serving a crucial role in the project, the UNC Health Sciences Library provided the students with the Muse headbands as well as guidance and advice throughout the process. “The library is an interdisciplinary connector,” says Moynihan. “It was a constant partnership throughout the Brain VR project.”
The UNC Health Sciences Library loans devices to students, faculty and staff, including activity trackers, sleep trackers, and motion control devices. For example, if a doctor with UNC Hospitals wants to learn how their patients are using a Fitbit, the doctor can check out a Fitbit to become more familiar with the technology and consider how to improve care for their patients.
Moynihan is hoping to build upon concepts from the Brain VR project. “If you can distinguish between those two brain states of moving up and moving down, you can do anything,” says Moynihan. “Wouldn’t it be great to control items in your home such as changing your TV channel just using your brainwaves? It also allows for more serious applications, like giving a paralyzed person the ability to communicate.”
To foster faculty, student and staff interest in AR/VR, Moynihan has created a campus AR/VR Interest Group for research and curricular applications of VR. The informal interest group works to promote innovative teaching and learning, research, training, clinical care and entrepreneurship through the use of immersive technologies.
For more information about the project as well as to access the open source code, visit the Brain VR website.
Story courtesy of Innovate Carolina
