Virtuality is a technology that has been tried and tested for many years, especially in the film sector. The use in other productions has always been very limited, because the image had to be rendered live for TV studios, which was not possible due to the long computing times of the rendering machines. Much has been developed in this area in recent years, and with the help of the actual gaming software Unreal Engine from Epic Games and various interfaces for implementation in everyday studio life, for example with Zero Density’s “Reality Engine”, a high level of photorealism can now be achieved, despite real-time rendering.
For studio productions, this opens up new possibilities that offer many advantages.
Virtual studios enable professional productions in a much smaller space than would be possible in conventional studio construction, and are also more cost-efficient and significantly more flexible, as a wide variety of formats can be produced in one studio.
By using additional robotics, the creators can also work much more flexibly, and production gains in terms of reliability, repeatability and thus also optical and workflow-based quality.
KST operates a virtual and XR studio with integrated CamBot.system in the Innovation Center where customers and interested parties can experience an exemplary automated and robotic workflow.
Components
Unreal Engine
Unreal Engine, developed by Epic Games, is a rendering and physics engine for video games. Since modern video games have almost the same use case as virtuality in the studio world, namely to render as photorealistically and live as possible, the Unreal Engine is also used as a basis for studio graphics systems. The Unreal Engine can be used to simulate realistic rooms as well as entire outdoor worlds with flora and fauna and realistic lighting and weather effects.
Zero Density
Zero Density provides the interface between Unreal Engine and the studio environment. With the product Reality Engine, Zero Density has created a studio graphics system based directly on the source code of the Unreal Engine, which is easy to handle but offers maximum performance and flexibility. It is only with Reality that the Unreal Engine can be used easily, quickly and in a sensible workflow for the studio world.
With the Reality Keyer, Zero Density has created a unique and revolutionary keying technology. Unlike the normal chromakey, this difference keyer does not work with a green color palette that is cut out, but with a reference image. Under optimal conditions, this method enables a pixel-perfect key without loss of detail such as, for example, in the case of a key that is not visible on the screen. individual hairs.
Another important core technology of the interface is the application of camera tracking data to the graphics to synchronize the perspective of the virtual and real world. Here too, Zero Density performs absolutely essential work by implementing all common protocols and calibrating the camera optics.
KST Moschkau GmbH
As a system integrator and project house, KST Moschkau GmbH is in a position to advise the customer optimally on his decision and to carry out the project completely from one source, from basic planning to the turnkey studio, including user training. In such an overall project, all subdivisions of the Moschkau group of companies work hand in hand to achieve the perfect and individually tailored result for the customer.
The strategic partnership with Zero Density and Epic Games not only enables KST to distribute across Europe and provide certified training. The KST Innovation Center also makes it possible to actively develop the workflow and the product.
Fields of application
Virtuality can be used in a variety of ways, and a green screen studio is only the most classic use case.
Augmented Reality
In contrast to pure virtual reality, augmented reality does not cut out and replace the background, but embeds individual virtual objects spatially and perspectively correct into the real environment. These can be photorealistic complex objects with shadows cast, or simple graphic objects, such as bar charts, which build up in front of the moderator. As a rule, no green area is required for augmented reality, which allows flexible use of the graphic elements in any conventional studio situation.
Virtual Production / Portal Production
Virtual production is a special application of virtuality. In this case, one wall in the studio, in complex cases even all walls, including the ceiling, is filled by LED walls or a large-screen projection. Instead of cutting out the presenter as in a green-screen studio and placing him or her in a virtual background, the virtual world is now placed directly on the screens. It is essential that the perspective of the virtual world matches the camera used. So from the camera’s point of view, the walls disappear and they become “portals” into the virtual world. For the external viewer, this distorted representation may feel confusing. With this production method it is possible to bring the virtual environment into real reflections in the action area and also realize the complete lighting of the set via virtual lights. This is how, for example, advertising clips for cars or even the series “The Mandalorian” are produced.
Hybrid applications
Of course, all described applications can be combined in any form to hybrid applications. From software used, workflow and design, the individual application areas do not differ. Depending on the situation, it may therefore make sense to combine green area, projection or LED wall.
Core technologies
Keying
Keying describes the releasing of the moderator or objects in the Green Area in order to use them in another environment. It is the most important technology in the virtual studio.
Zero Density’s reality keyer is a difference keyer and does not work with a green color palette that is cut out, but with a reference image. Under optimal conditions, this method enables a pixel-perfect key without loss of detail such as, for example, in the case of a key that is not visible on the screen. individual hairs.
Three-dimensional virtual design
One of the most important components of virtuality is three-dimensionality. All virtual objects and also the environment are three-dimensional. This allows both the camera and the presenter to move freely in this world and around these objects. The three-dimensionality is essential for a correct perspective and general realism. The design side is given simple tools by using the Unreal Engine and there are almost no limits. Thanks to the large community and the Content Marketplace, it is possible to obtain expertise for little money and use already designed objects or environments.
Green area and lighting
The main component of a classic Virtual Studio is the Green Area, an L- or U-shaped fillet painted in a special green color that is as homogeneous and low-reflective as possible.
The Green Area only determines the range of movement for the presenter and physical cameras, but has nothing to do with the final visible size of the virtual space, so the Green Area can be significantly smaller than the virtual studio shown later.
The correct illumination is the second important part for a shadow-free and even green that allows an optimal key. Adjustable light only needs to be used in the Zero Density workflow to provide additional illumination or effects to the presenter. The optimum basic illumination for the keyer is daylight that is as homogeneous as possible, i.e. luminaires with a temperature of approx. 5,500 K.
Tracking
Camera tracking is essential to enable camera movement. Tracking in this case is a data stream containing all spatial and rotational coordinates of the camera. Also all data about the current state of the optics are included in this stream. With this data, the rendering machine is able to synchronously adjust the perspective of the virtual camera to that of the real camera at any time. For the illusion of a virtual studio, precise tracking is mandatory.
But tracking is not only used for camera tracking. With Talent-Tracking it is also possible to track the moderator and his movement. This allows the moderator to move naturally in the virtual world and interact with virtual objects.
Rendering machines and composition
Most of the workload in the graphics workflow is carried by the so-called render engine. This is a high performance workstation that is responsible for one rendered camera image at a time. For performance and redundancy reasons, it is recommended to use one render engine per rendered camera strand.
A particular advantage of the zero density workflow is that both the keying process and the rendering of the graphic take place on the same machine. This realtime composition allows the software to coordinate internally, providing realistic interactions between camera image and virtuality.
Live Operating
The interface to the control room in live operation is also important. With the profound software and hardware systems Reality-HUB, Panasonic KAIROS and CamBot.system, it is possible to configure the entire system down to the smallest detail from a software interface and to handle, control and fill it with content in live operation by means of an interface prepared for the operator. Connections to automation systems and databases are also possible.
Contact
Various tracking options for virtual productions
21. November 2023Set construction in the traditional sense is no longer used in many film productions and television programmes, or only to a very limited extent. VR makes it possible: instead of having to build elaborate sets, the shape, colour and size...
KST Weather Control Module for Zero Density’s RealityHub
21. March 2023The KST development team has recently completed the “KST Weather Control Module”. It is an interface for Zero Density’s RealityHub and can be integrated via API. The weather module automatically loads the current weather of the city selected in the...
IBC 2022 – Virtual studio production automation
8. September 2022At the Zero Density booth, KST Moschkau showed the extent to which productions in virtual studios can be automated with Reality, CamBot systems and Kairos. As a partner at the Zero Density booth, the KST CamBot team gave deep insights...