Under the hood of virtual production

Virtual production (VP) is a term that encompasses multiple technologies across several stages of film, television and media production. It gained momentum during the pandemic because it offers a number of advantages over traditional studio technology.

“The pre-production phase involves tools for storyboarding and concept design, as well as for previsualisation, the visual arts department and location scouting,” explains Vincent Steenhoek FRSA, co-founder, CEO and Senior Consultant at VP company Framesync. “We see VR and AR increasingly used for both storyboarding and previsualisation, where it automates and enhances the process, offers quicker iterations and greater creative freedom and allows teams to walk through virtual sets and plan scenes in detail before actual production.”

At the production phase, LED walls and camera tracking technologies are core to virtual sets and LED volume stages. “They have revolutionised the way environments are created and filmed, offering a dynamic and immersive backdrop that replaces traditional green screens,” says Steenhoek. “The ability to shoot with actors and physical objects directly in front of virtual environments displayed on massive LED screens has dramatically reduced the need for green screens and post-production visual effects (VFX), allowing for more real-time collaboration and decision-making on set. It also offers more natural lighting, as the LED screens can emit light consistent with the environment being displayed, improving the overall realism.”

These tools work alongside real-time rendering products and motion capture (MoCap), camera tracking and simulcam. “AI advancements in motion capture allow for more precise tracking with fewer markers, reducing costs and increasing efficiency”, he adds.

At post-production, the focus is on compositing and VFX, alongside editing and sound design. Finally, Steenhoek highlights the importance of cloud collaboration and asset management in VP. “The pandemic accelerated the need for remote collaboration, and cloud-based tools enable teams to work together from anywhere, streamlining asset management and review processes,” he says.

Steenhoek notes that the integration of real-time rendering engines like Unreal Engine has transformed the speed and flexibility of production, allowing filmmakers to visualise and adjust scenes instantaneously. This blurs the lines between pre-production, production, and post-production, creating a more fluid and iterative workflow, a trend also picked up by Stan Moote, CTO of IABM.

“VP brings post and VFX forward in the shooting timeline, in-camera visual effects (ICVFX) for example are now used in a range of workflows,” says Moote. “The old ‘three stage’ notion of pre-production, production and post-production is very much blurred in a VP environment.”

The integration of game engines with the creation of media and entertainment is causing the convergence of two traditionally separate technology sectors and enabling the crosspollination of talent and techniques, according to Moote. “The broadcast media tech sector will benefit from attracting talent from the games industry through collaborations that are initiated by the development of VP projects,” he explains.

The integration of traditional 'post-production' activities’ into the production process is another positive change for Moote: “There are benefits in terms of efficiency of course, but the creative flexibility is much greater. The instantaneous way in which a director can see the world and the performances with it are changing the way film makers work.”