The Challenges of Adapting Cinematography to VR

There was a question today on the Oculus VR Developer Forums about cut scenes in video games and how to approach them in VR.  Essentially, the original poster was asking how to address the conflict between cut scenes, which typically follow the language of cinema, offering frequent cuts from one point of view to another, and VR where 'cuts' can be disorienting to the viewer.

Here's my response, only lightly edited:

This is an interesting topic and is part of understanding the intersection of VR and cinematography as a whole. Part of cinematography is the manipulation of the frame, the field of view, the depth of field, the lighting, etc in order to convey an emotion or idea.

However, the language of cinema doesn't necessarily translate 1:1 to VR. Sudden transition of the viewpoint in film, or 'cuts' are so common in that it's considered notable when you go for long periods without them. But as you say, they can be extremely disconcerting when done in VR, where you've increased the level of immersion specifically to make the viewer feel more like they're present in the scene than simply a viewer. This means that common things like the 'Angle / Reverse Angle' sequences used to alternate between closeups of two people speaking as they converse is something that doesn't really work in VR.

The problem is, how do you convey certain non-verbal ideas without saying them. Cinema itself had this very problem, because it was a medium that grew out of the effect of disruptive technology on a prior storytelling medium, the stage, which in turn had to deal with the same problems.


Consider two people trying to communicate privately without other parties nearby hearing. For instance, this exchange from Romeo and Juliet, Act I, Scene 1:

SAMPSON: [Aside to GREGORY] Is the law of our side, if I say ay?

GREGORY: No.

It includes the stage direction Aside to GREGORY indicating to the actors that this is intended to be a private question asked of one person to another and not heard by the other people in the scene. In stagecraft, this can be done with exaggerated body language, where you rely on an actors skill at conveying emotion and ideas with his body as a whole, because the audience doesn't necessarily have a good view of the actor's faces. In cinema, the point of view is no longer fixed, so you can convey the idea by actually cutting from a wide view of the scene to a shot that only includes these characters, close enough to convey a sense of intimacy. This is then close enough to allow the actor to whisper and yet the viewer from this new perspective can plausibly hear the exchange.  This not only changes the technique but also the skill set required from the performer.

Stagecraft solves the problem one way, cinema another, both suited to the constraints of the medium. Before new techniques were developed and refined, most cinematography was essentially nothing more than filmed stagecraft. VR will have to find a new way. I suspect that for a significant while, cut scenes and the like, when ported into VR will end up simply reusing the language of cinema, and have to find ways of coping with trans-location.

The easiest is to find a way to transition to a more screen like experience for the duration of the non-interaction.  Perhaps the players view could slowly reduce in FOV and depth until they were looking at a more traditional screen shaped window of video that hovered in the center of their vision before the transition to the preset content.  This provides a slow reduction of the immersion to the point where you can now present cinema-like content without the corresponding disorientation as a viewer feels like he's being teleported repeated and not allowed to shift their point of view.   Alternatively there could be some kind of screen or display in the environment to which their attention is somehow drawn or even forced (although involuntary orientation changes can be just as disconcerting as involuntary trans-location).

Simply playing the cut scene around them in the way you suggest with the 'ghost' idea might work, but it largely depends on what information is being conveyed. In such a mechanism you still have to have a way of drawing attention to small details that are important to convey, although this might be possible with good sound editing and and doing triggered events rather than a fully-non-interactive cut scene, but that in turn might depend on knowing you have really good positional sound available to the viewer, which the Rift itself doesn't guarantee.

The real answer is that there's no right answer, nor even an established set of techniques meant to act as a VR vocabulary of ideas in the same way that there is for cinema. It's a good question to ask, but in addition to asking forums like this for ideas, you should work on developing your own. Who knows, you might come up with something so innovative and groundbreaking that it becomes the 'Citizen Kane' of VR.