I worked on both the design and implementation of the core interactions that we presented to Disney Imagineering R&D. As well as the fundamental systems for achieving this technical exploration.

One of the major systems I developed was one to handle skeletal tracking inside of Unreal Engine. We used multiple Azure Kinects to capture skeletal data and passed that into the engine, my system stitched that data together and provided clear and accurate positions of users in the space, as well as uniquely identified each one. This required a good understanding of the skeletal data we were working with but also of 3D math to be able to transform the data into usable and correct spaces.

Another system I was responsible for creating was a gesture recognition system. This system was able to take in a series of user defined positions in time frames and associate that with output events. This allowed us to register certain actions like a wave or point and pass that event into any actions we desired. You can see an example of this in the video.

Not in the video are three important systems that I developed for this exploration. First was using NDisplay to run the application visually on an LED volume using the needed focal settings to achieve a realistic effect. Second was creating a generative mesh of the person inside the scene based on the skeletal data to create a realistic shadow and collision profile. And lastly was using the skeletal data and other hardware based inputs to drive physics objects in scene and react on stage with physical props. The best example of this was a physical rope that lifted and reacted to a physics object attached to it in the virtual scene.