Microsoft HoloLens Technology Puzzle (bottom)

The reader asks "HoloLens's depth sensor is it possible to be based on TOF?" ”

First of all, there are three types of depth sensors that are common on the market:

• Structured light, this technology represents the product of the Kinect generation, and its sensor chip is used by the PrimeSense family. By the way, PrimeSense is now Apple's company, and the field is sure to be exciting in the future.
• Tof,time-of-flight, on behalf of the product is the Kinect II, because Microsoft's love for one word, its official name is Kinect one, a bit confusing, right?
• The binocular camera, on behalf of the product is Google Tango and Leap Motion, the former with four cameras, the latter two.

I think the reason why HoloLens didn't use the TOF technique is because if the four cameras are placed on a tof sensor, a forward-facing RGB camera is missing for the video chat App. The reader mentions the middle part I think is miniature projector, HoloLens use micro-cast on "windshield" to display holographic screen. See Wired's report http://www. Wired.com/2015/01/microsoft-nadella/

As for the answer to this question, we have to wait for Microsoft to release new information.

The reader asks "can HoloLens be used as VR?"

This is a good question, and indeed it can be played, NASA's Mars cooperation program with Microsoft is basically a virtual reality application that ignores the real environment. If the HoloLens can provide an API to adjust the "windshield" transmittance, then it is like wearing a Oculus Rift helmet when fully opaque, which is another way to experience VR. But friends, this is a waste of HoloLens function Ah!

So, as an application developer of the future, how should it be fully drained of its function? This is the focus of this article.

Body part

First of all, let me have the brain for a while, for everyone "Introduction" under the Holo SDK. According to Microsoft's preference, the SDK standard language must have C + + and C #, so to develop, these two languages you will have at least one door.

Then, what are the features in this SDK? According to the scene in the official demo, I think the basic features are at least:

• (a) The image that the camera sees, that is, the color buffer of the current scene.
• (b) The depth map of the current scene, depth map or Z buffer.
• (c) SLAM synthesized three-dimensional scene, this scene of the space below we are called holo space, it may be in the form of Lego block, or may be represented by a triangle.
• (d) coordinates of the HoloLens device in Holo Space (x, Y, z), direction (TX, Ty, TZ).
• (e) The result of gesture recognition, similar to the appearance of HRESULT ongesturedetected (DWORD Dwhandid, DWORD Dweventid, LPVOID lpuserinfo).
• (f) The result of speech recognition, similar to the appearance of HRESULT onvoicerecognized (std::string& strsentence, FLOAT confidence).

Similarly, according to the official demo, I divide the HoloLens application into three types:

• The traditional application of pseudo-holography

• Specifically optimized for HoloLens applications

• Immersive true-holographic game

This article only speaks of "pseudo-holographic traditional applications"

This type of application is the easiest to use for traditional developers, with little need to modify the code, and naturally requires no knowledge of 3D graphics. Most people will start holo development from this application.

If you do not need 3D knowledge to achieve 3D interface, then the effect of 3D where? That's the Holographic window manager--explorer3d.exe introduced in Windows 10. We usually start Windows to see the "desktop" is a window manager (Explorer.exe) part of the "desktop" to think of three-dimensional on the line, very simple, right?

It

's not a simple thing!

In the case of a video player, the player does not directly display the video to the desktop, but instead draws it to a cache area, and after a series of steps that I do not know, Explorer.exe will then copy the screen "2D" to where we can see it. And the Explorer3d.exe uses the "3D way".

The so-called "3D Way" is to use Direct3D to do some "will people think I talk too much, no one will see or not" thing ...

That is, when initializing the application:

• Creates a 3D rectangle to represent a 3D window, saved in the vertex cache vertex Bufer (which can be thought of as an array on the video card)
• Create a decal texture (can be thought of as a picture on the video card)
• Create a Local_matrix that represents the window's translation, rotation, and stretch values (The matrix here is the concept of matrices in mathematics, not the movie name)

While the app is running:

• If you need to move a window in Holo Space, then modifying the translation values in Local_matrix is similar to moving windows in Explorer.exe, except that you can move forward and backward (z-axis) in addition to the upper and lower (y-axis direction) movement around (the x-axis direction).
• If you need to rotate the window in Holo Space, modify the rotation value in the Local_matrix. There are no similar features in Explorer.exe.
• If you need to indent in Holo Space, then modify the Local_matrix in the Explorer.exe, and the function is consistent with the drag and drop of the edge of the window at the same place.
• If the video content needs to be updated, then the update texture is the newest content, only then we can see the moving video.

When the app exits:

• Releasing Resources for Direct3D

Do not worry, these are explorer3d.exe will be responsible.

Attentive readers will find that we do not use a very important function:

(d) coordinates of the HoloLens device in Holo Space (x, Y, z), direction (TX, Ty, TZ).

To explain it I have to talk about the world, view, projection three matrices, but also to explain the matrix multiplication and dot product formula and so on, these are completely graphics knowledge, 1:30 will not finish, I will write a series for them again. I'm going to explain in a nutshell that Explorer3d.exe will maintain a global global_matrix, which equals the value of the world * View * Projection,view is obtained by the information in function (d). Global_matrix role is like the first person shooter game Mouse, wrote here I found that Explorer3d.exe as a 3D game will be a lot easier, 3D game scene TV can play video, in fact, the same truth.

To summarize, this type of application programmer almost does not need to modify the code, the Explorer3d.exe is responsible for calling Direct3D to convert the traditional application content into a map.

Conclusion

This article is first written here to see the feedback from everyone. Although the article is called "HoloLens technology Puzzle (bottom)", but rest assured that this series has not ended, with the gradual development of the brain hole, I found to introduce more and more content.

Microsoft HoloLens Technology Puzzle (bottom)