Mirrorsys: What kind of experience is immersive reality?

Dr. Wen/james Begole

If you have been fascinated by the plot of a novel, you may not need any visual detail to convince you that you believe it, because the story itself will bring you to the scene. The same is true of video games, such as Minecraft (My World), a video game with a low-fidelity block of data that brings the experience to the most advanced, photo-quality game. Low-fidelity mobile games have been deeply into the high-fidelity gaming machine Market, the main reason is that people can be immersed in the game anytime and anywhere. However, studies have shown that visual and auditory details do affect our perception of reality and make us more convinced of reality, while realistic visual and auditory effects can make us more immersive, as if we were in the story as an integral part of the story.

What is an immersive experience?

The game is probably the most iconic killer application of immersive reality (Immersive Reality). In the game, the player wears headphones and swims in a man-made world. However, virtual reality is not confined to the artificial world generated by the computer, it also includes allowing the distant scene to become near. By deploying high-definition cameras, microphones, and low-latency networks, our eyes and ears perceive light and harmony from distant regions, making this all true. This kind of remote reality is a kind of virtual reality. All physical objects in remote reality are real in the real world, but not in front of you.

Now the camera has an amazing high resolution. Typical consumer camera pixels are over 12 million, and even some image sensors have more than 120 million pixels. At the same time, the display resolution is increased from HD (2 million megapixels) to 4K Ultra HD (8 million megapixels), and the resolution of the new display screen can reach 8K (32 million pixels). Maybe when we watch TV or movies with ease, these resolutions are more than enough. But when we're in a remote space for communication, are these resolutions enough to keep track of our goals and adjust the focus to create an immersive experience?

Traditional movies and television show The audience is a "back-up" experience, the audience can only passively listen to others to tell stories, or follow the camera's lens movement and move. However, in an immersive experience, each audience can define their own points of interest, and their points of interest can be changed at any time depending on the activity scene. What's clear is that they are recreating reality. It's not as if you're directing the audience's attention to a character or object in a scene, as in a movie or a story. In immersive reality, all the visual and auditory data an audience wants must be ready for the moment they turn their heads.

Immersive remote reality opens up a new interactive experience in which rich visual details have a huge impact. Imagine surgeons around the world who can operate remotely, skilled technicians who can remotely manufacture and repair complex machines, and students can reach the most cutting-edge science on earth and eventually move to a vast expanse of Earth. People will soon be able to travel to exotic countries by flying drones, manipulating robots to talk, shopping in the world's most exclusive boutiques, watching world-class athletes on the ground, and feeling the thrill of live music on stage with bands – all in real time with a low-latency network.

The value of these experiences is clear, but there are still some key issues with the development of related technologies, such as: What are the insurmountable limits to maximizing the value of immersive experiences? Once this question can be answered, we will be able to know the maximum amount of data that cameras, monitors and networks need to handle.

The upper limit of human visual perception

It turns out that even today's most advanced cameras and displays can only represent the scales of the world around us. The point resolution of the human eye is 0.3 cents, which means that the human eye can distinguish nearly 200 distinct points at a glance. If we convert it to a "pixel" on the screen, it depends on the size of the pixel and the distance between the human eye and the screen. We make reasonable predictions at 200 pixels per degree. The human eye can mechanically move the span of the horizontal 150 degrees, perpendicular to 90 degrees. To fully cover this span, 540 million pixels are required.

Just one static screen requires up to 540 million pixels. However, our real world will not be static, multiple static images in sequence to form a dynamic video, the current film and TV generally flash 24~30 frame screen per second. However, the human eye does not work like a camera. In fact, the human eye receives a continuous picture of light, not intermittent. For movies or TV shows, 30 frames per second of dynamic video is enough. But the human eye can perceive faster motion video, which is expected to be up to 200 frames per second. For sports, games, research and other high-speed immersive experiences, this requires at least 60 frames per second or even 120 frames, which avoids the blur or dislocation of the dynamic image.

In addition, there are other features of the human eye and the current display technology can not be a candle. For example, the human eye can perceive a contrast of nearly 1 million luminance levels, which requires up to 8 bytes to fully encode the gamut of each screen pixel.

Now, let's take a quick, rough calculation of the upper limit. 540 million pixels, 8 bytes per pixel, 120 frames per second, which requires 518GB of data per second. Not only now, but even in the foreseeable future, no data system or network can have such a large raw throughput. Fortunately, visual data redundancy can achieve a large amount of data compression based on the complexity of the screen. Even with a compression ratio of up to 300:1 (which enables this compression ratio to be encoded by a very powerful computer system), 518GB data streams also allow us to stop at 1.7GB of data per second. 1.7GB of data per second sounds huge, but it's just the fur of our peripheral data in the real world. With our head and body turning ability, we can extend the field of vision to 360 degrees, perpendicular to nearly 270 degrees, which is the range that the human eye can focus on at all times. Covering this range requires 3.8 billion pixels, if 120 frames per second, this also requires a 3.7TB per second transmission system! Because the human eye can adjust the focal length at any time, thus enriching the characteristics of human vision, these features at the same time to the various needs of technology to new highs. In the future, a monitor with a focal length adjustment function needs to be able to bring an object into or out of our sight dynamically. This is not possible today for commercial 3D displays. However, the light field display in the research lab can do it, but more data is needed per frame.

All-round communication is not just about transmitting massive amounts of data, it also requires compressing and transmitting the data in real time. As early as 1968, Robert B. Miller, an early human factor scientist at IBM, argued that the threshold for human-defined "immediate" response was within 100 milliseconds. The vertigo that people suffer in virtual reality systems is caused by the delay between human dynamics and systemic reactions. To slow down this type of vertigo, you need to shorten the delay.

Let remote reality become reality

In today's world, most home networks have a bandwidth of no more than 20M and a commercial network of no more than 100M. The massive amount of data mentioned earlier is only 1% of the data throughput in immersive reality. You may wonder if this super-fidelity requirement really needs to be, because HD video is just replacing SD video in the market today.

But do you still remember that when HD came out, it seemed like a future-oriented format. Now, however, it has quickly been replaced by ultra-high-definition video cameras and displays, with a 43-inch 4K display priced at less than $600, and many video streaming services have begun to provide 4K content.

The resolution and physical dimensions of the display are also evolving as several cameras and display manufacturers have released 8K products this year, and some data flow services are beginning to provide 8K content. Once upon a time, the new playback standard took more than 10 years to be widely used, but the ultra-high resolution system is not the same, the ultra-high-resolution system is now being used quickly, because they can be downloaded on the Internet on demand.

Mirrorsys system for video and environment integration

Mirrorsys: Network is kingly

The first step in Huawei's omni-directional communication is the introduction of a research prototype called Mirrorsys, which mirrorsys real and real-time reproduction of remote space in the same size as the physical size from both visual and auditory aspects. Huawei has shown high-profile Mirrorsys at the 2015 Mobile World Congress in Barcelona and the Hannover Information and communication technology show, which includes a 32 million-megapixel, wall-sized display that is seamlessly arranged by 16 HD projectors, 5 meters wide, 2.6 meters high, fully conforms to the human retina two meters distance of sight resolution. Following these early demonstrations, Huawei has made a number of new developments in the lab, including:

• The width of the display is enlarged by one-fold, that is, the display is 10 meters wide, 2.6 meters high, and pixels are enlarged by one times;
• The audio system uses 32 microphone arrays to capture the directionality of the sound in the source environment and is reproduced accurately through a 22.2-channel speaker system. Since the system accurately locates the origin of each sound, listeners can distinguish between simultaneous conversations in different areas of the shared space;
• 3 x 4K Ultra HD cameras are stitched together to run at 60 frames per second.
• Compression of loads enables fast transmission within 150 milliseconds through a dedicated network. There is also a continuing commitment to creating the lowest possible delay.

While the first generation of Mirrorsys prototypes can push today's network infrastructure to the limit, it is only the tip of the iceberg that touches the full range of communication technologies. Whether the source image is from a real-world camera or created by a computer, Huawei knows that a truly immersive experience requires massive amounts of data to match the sensitivity of human perception. The future network needs to be able to transmit a larger order of data at low inter that humans cannot perceive.

Through all this, Huawei understands that ultra-high-speed networks are essential for the wide application of full-immersion media. In addition to playing the game on a local dedicated machine, most applications will need to transfer some or all of the "reality" between the server and the end user, and even the game is being migrated to the cloud infrastructure. Cloud infrastructures need to transmit large volumes of data around the world, and data center infrastructure is becoming increasingly important for presenting a real human world.

Huawei's research and development department is stepping up the development of technologies capable of compressing, transmitting, and reproducing light and sound in real-world remote environments or virtual environments, in real-size, full-fidelity ways.

mirrorsys, or the future of omni-directional communication, is the ability to bring people to any place in the world regularly and truly." The attainment of this achievement will open up a lot of new business opportunities. For example, the original telemedicine was medical communication through high-definition or 4K ultra-high-definition video lines, and the omni-directional communication technology allowed them to perform accurate medical diagnosis and remote surgery under comprehensive visual details; technicians can remotely operate and repair complex machines Shoppers can view real estate, gems, detailed manufacturing processes, electronic circuits, and other visually complex products with unprecedented precision. For consumers, the full immersive system allows them to participate in areas beyond their wildest dreams, such as climbing Everest, visiting the Taj Mahal, driving formula cars, watching a World Cup game, skydiving, or diving on the Great Barrier Reef. All this is both spectacular and inspiring. Even if you are in a faraway place, you can visit your mother remotely and check your child's bedtime before going to bed, which will also be a wonderful moment in our lives. Augmented virtual reality is opening up countless new opportunities-but how many realities can a person exist? Of course, there is only one, but it is full of infinite possibilities.

(For more information about Huawei, please follow Huawei's developer community, Huawei's own open door: http://developer.huawei.com/cn/ict/, don't ask me what I call, others call me Lei Feng )

Mirrorsys: What kind of experience is immersive reality?