Microsoft's latest patent describes how to project a panoramic image in a room

Absrtact: Microsoft's latest patent describes how to project panoramic images in the room (Tencent Technology map) A newly released patent shows that Microsoft may plan to extend the game to the TV display, by projecting an external image across the room

Microsoft's latest patent describes how to project panoramic images in a room (Tencent technology map)

A new patent shows that Microsoft may plan to extend the game to a TV display, by projecting an "external image" across the room, providing a 360-degree view of the virtual scene.

Microsoft's patent, the "immersive display Experience" (immersive display experience), was announced last week by the U.S. Patent and Trademark Office (USPTO), which was first filed in early 2011. The patent describes a standard video game system with an external "environment monitor" that can project panoramic images in a "Look around the user" way.

Such a projector would not replace a TV display for the current game console, but would provide an "external image" as the "extension" of the primary display, with the aim of expanding the gaming environment beyond the TV display so that the player could "turn around and watch the enemy sneak up from behind."

The display device described in this patent is tightly integrated with a deep sensing camera system and can be encapsulated in the same outer packing box as the ambient projector. This device may be a standard dual-camera structured light sensing system, like the current Kinect, it could also be a more sophisticated device that incorporates "multiple image crawlers" to "stitch a panoramic image using multiple captured images" and project it across the room.

Regardless of its form, the depth sensing camera system described in this patent can assist in environmental projectors, the system can perceive the layout and topography of the room, allowing the projector to provide features such as color and distortion correction, so that even when the image is projected onto different walls and furniture surfaces, Does not seem to be distorted.

In addition, this environmental monitor can also use the player location information provided by the camera system to "adjust the visual panorama of the external image display" to ensure that the environment still looks natural when the player moves in the room. The patent even describes creating a "protection" area where the player is standing, so that the light from the projected image does not directly illuminate the player's eyes, even when the player is looking toward the projector direction.

The projection image described by this patent "may be displayed at a lower resolution than the primary image", but the patent document states that it will not "adversely affect the user experience". The projected image does not necessarily show a planar state on the wall surface, players can wear stereo 3D glasses to gain an extra sense of depth and realism, which is somewhat similar to the concept of "Fortaleza glasses" smart glasses, as mentioned in the Microsoft document leaked this June. This is an expanding reality technology.

How the projector detects the user location and creates the "protection" area (Tencent Science and Technology match map)

The game system described in this patent may never turn into a full-featured product, but such an environmental system could bring real change to people's way of playing games and thinking about video games. In my opinion, one of the main factors that restrict the movement control is that the player needs to face the TV display screen to know what is happening in the game, and the result is that in many games, the player can only use the action of tilting the body and moving to the side to replace the turn or move in parallel with the display.

However, if the game image can be projected onto the four walls of the room, then the user can actually move across the room and look in any direction so as not to miss what is happening in the game. Imagine a new version of the Dance central game, which will show the Asser Moures (Arthur Murray) footprint on the floor, or imagine an adventure game in which players need to explore the whole room to find hidden treasures; Or to imagine a first-person shooter, the player can turn around in reality and face the muzzle that is projected on the right side of the body, not just the response to the red flash that appears on the right side of the display.

If the environmental monitor really works as advertised, a head-wearing virtual reality device like Oculus Rift is no longer necessary, using sophisticated head-tracking techniques to simulate the surroundings of the player.

However, it is a bit of a concern that a 360-degree projector can really change a room. Unless the projector described in this patent is mounted on the ceiling, I cannot understand how it can avoid the big problem of image shadowing-not just by the body of the player, but by the furnishings and other items in the room. Color and image distortion correction is a good idea, but if part of the scene is supposed to appear in the corner of the room and it's in front of the corner, it's always a little unsettling.

But it is always interesting to see how Microsoft's research and Development department is considering how to extend video games to the black borders of the TV in a revolutionary way.