Imagine if you could enter a play by putting on a virtual headset. Now imagine that the characters in the play are shaking your hand or giving you a hug, and that you can feel them do these things.
This is the kind of experience that researchers at 黑料吃瓜资源 are developing through the 鈥淢eta-Physical Theatre: Designing Physical Interactions in Virtual Reality Live Performances Using Robotics and Smart Textiles鈥� project. In a nutshell, the project integrates physical touch into virtual reality (VR) live performances.
is the lead researcher on the project. Dr. Pan is an Assistant Professor in the Faculty of Engineering and Applied Science and a member of Ingenuity Labs Research Institute at 黑料吃瓜资源. He was one of six inaugural recipients of a in 2024, supporting community-focused research that pushes boundaries in technology and society.
Virtual Reality Beyond the Visual
In association with intersectional arts organizations, this project aims to build immersive narratives where participants can not only see and hear virtual characters but can also physically interact with them. It pushes boundaries on VR environments to build immersive environments where touch becomes a part of the narrative structure.
Dr. Pan鈥檚 idea began years earlier during his time at Disney. While working on Star Wars: Galaxy鈥檚 Edge, he developed an immersive experience where visitors could feel an iconic 鈥渇orce grab鈥� (when a Jedi summons a lightsabre through the air). 鈥淵ou would put on a VR headset, and you would see, in the distance, this lightsabre that you can reach out to with your hand and it would start zooming toward you,鈥� he explains. 鈥淵ou would actually see the lightsabre come into your hand in VR. At the same time, a robot in the real world would deliver a lightsabre prop with the exact same timing and force.鈥� Though the project was ultimately shelved by Disney, Dr. Pan didn鈥檛 give up on the idea. 鈥淚 thought there was a lot left on the table by shelving that project.鈥�
Making Touch Feel Real in VR
Of course, there’s no VR theatre without theatre, and Dr Pan’s collaboration with is essential to the project.聽 Wheeler is a fellow Ingenuity Labs and Connected Minds member, Assistant Professor in the DAN School of Drama and Music, and Director of Artistic Research at SpiderWebShow Performance. “Shortly after arriving at 黑料吃瓜资源, I was introduced to Michael鈥� we thought it would be really cool to actually have a theatrical narrative that uses interpersonal interactions in VR,鈥� Dr Pan says. Supported by community organizations, Dr. Pan and Wheeler co-created a live VR theatre experience that integrates physical touch. 鈥淚t鈥檚 a high risk, high reward project that Connected Minds was willing to fund.
鈥淸We are] creating this narrative that involves physical interactions with virtual characters. [We are] starting out simple, we鈥檙e looking at simple interactions like high fives, or fist bumps, and handovers of objects where you don鈥檛 necessarily need a lot of fidelity in terms of physical interactions.鈥�
To make these moments feel real, the team uses haptic proxies. As Dr. Pan explains, haptic proxies are physical props that 鈥榮tand in for haptic interactions you would normally feel in the real world.鈥� For example, a robot-mounted hand can simulate a high five at the exact moment the participant sees it in VR.
However, matching physical actions in the world and virtual actions in VR creates a major technical challenge. The system must align spatial coordinates using motion capture and high-fidelity 3D pose tracking, so that the location of the proxy in the real world matches the location where the VR headset thinks you should be.
Timing matters too. The team must also synchronize physical and virtual actions on the scale of milliseconds. 鈥淔or dynamic experiences, it鈥檚 even more complicated,鈥� Dr. Pan explains. 鈥淧articularly for handovers or high fives, there needs to be not only a physical correlation, but also a temporal correlation. You can鈥檛 have the high five happen in VR first, followed by it happening 500 milliseconds later in the physical world. It breaks the illusion.鈥� To avoid this lag, the team uses a system that shares information between the VR environment and robotic devices to keep latency low and synchronization precise.
Collaboration Across Disciplines
The project is supported by two arts organizations: and . The former is a Kingston-based arts organization and Canada鈥檚 first live-to-digital performance company. It focuses on exploring the intersection of live performance and digital technology. 鈥淲ith SpiderWebShow, we work with Adrienne Wong, who is contributing to the dramaturging鈥� says Dr. Pan. bCurrent is a Toronto-based company that supports the work of Black and intersectional artists and plays a role in shaping the narrative voice of the theatrical experience. Together, these collaborators ensure that the narrative experience is inclusive and culturally relevant.
For Dr. Pan, it鈥檚 important that the creative process among engineers and artists is authentic. 鈥淲e are emphasizing the co-creative nature of this project … [Michael and I] talk about these experiences at length and we have many ideas on what we eventually want to do with this technology, but one of the most important steps is we鈥檙e not leaving each other out in the dark.鈥�
Beyond Theatre: Next Steps
Dr. Pan has big ideas on where the project and technology could eventually go.
鈥淲e already have inquiries into sports training,鈥� he says. 鈥淭here鈥檚 lots of implications for being able to customize training regimens for athletes.鈥� For instance, being able to train a hockey goalie in a safe and replicable environment without needing live opponents or expensive setups would be helpful to coaches.
The technology could also support hands-on training for skilled trades, with the potential to lower barriers to technical training and to improve safety. 鈥淲e could use a robot to mimic a lathe, and then do operator training in VR, especially when there is a shortage of machine equipment or safety concerns, we could have novices training with haptic proxies before moving on to the physical machine.鈥�
Beyond performance and training, Dr. Pan is also excited about its application to care for the elderly and combatting loneliness. He鈥檚 been speaking to , a Connected Minds researcher at York University, who studies VR in palliative and geriatric care. 鈥淟oneliness is a huge issue with the elderly population,鈥� he explains. 鈥淸We are exploring] if we can use this technology to make social connections.鈥�
Conclusion
In a world where screens mediate much of our social world, Dr. Pan asks: what if we could reclaim a sense of touch, even through a headset? His project brings together engineering, art, performance and community, showing that immersive technology isn鈥檛 about tricking the eye, it鈥檚 about restoring presence and human connection.
