The final experience: A procedurally-driven storytelling engine and virtual reality experience, The Past Tense of Touch appropriates the visual language of 3D video games and the gestural language of cinema, combining the two in unexpected ways. Each time the piece runs, the setting and characters are different, drawing from a repository of stock 3D game assets. The characters’ gestures and related actions with respect to one another and the user are determined in real time, each time creating new narratives and plotlines. The user’s gaze, position, and gestures will have the potential to heighten tension between the two characters and incite different dramatic twists and turns in emergent plotlines. Drawing on a dataset of gestures from a set of films, in which the gestures have been removed from their context and applied (via motion capture data) to 3D model characters, each emerging narrative will inherently explore themes of power, intimacy, love (triangles) and expressions of violence.
The three phases of development:
Phase I: Transposing cinematic motion capture onto stock 3D models, letting full scenes play out between 2 characters (from two different films). The output is a non-interactive 3D scene to be experienced in VR so the characters can be tracked. However, there is no interaction component at this phase.
Phase II: Same as phase I but instead of a full scene, the individual gestures will be clipped, tagged and procedurally drawn and interpolated from one gesture to the next using an animation blend tree in Unity. This will also be presented in VR but will not have an interaction component.
Phase III: This phase introduces interactivity. The scenes from which the gestures are drawn will likely be limited to those involving a third character. Developing the parameters of interaction will require extensive testing, but the main inputs will be: gaze, position and orientation relative to each character.
Phase I: Story Design
An example: Mapping the movement and gestures of two scenes
Thelma and Louise scene 01:16m in when Louise is washing her face, sees an elderly man sitting about 15 feet away. She approaches him, they gesture hello to one another and Louise sits down on a crate next to him. She begins to remove her jewelry and, looking him in the eyes, hands it over to him. In this scene, she is active, he is passive.
Ex Machina scene in which Nathan shows Caleb his bedroom. Caleb takes a look around, Nathan stands there, stoic. He has Caleb sit down at the desk to read the NDA and then lies down on the bed. Caleb hunches over to read the NDA, he questions it. Then Nathan stands up, standing above him. Caleb is passive and remains hunched over as he makes the decision about whether to sign it. And he eventually reveals a feeling of excitement, his posture straightens, as he decides to sign it in order to have access to the development of the AI.
Scene-by-scene breakdown of scenes from selected films. Organized by if there are exactly two characters in the scene, and then by the gender of each of the characters.