HoloLens Development notes using unity to develop a simple application

I. Overview of HoloLens

HoloLens has the following characteristics

1. Spatial mapping
With Microsoft's specially customized holographic processing unit (HPU), the HoloLens enables fast scanning and spatial matching of the surrounding environment. This ensures that the HoloLens can accurately place or present holographic content on the real world surface, ensuring the core AR experience.

2. Scene Matching
HoloLens devices can store and identify environmental information, restoring and maintaining holographic objects in different scenarios. When you leave the current room and return, you will find that the original holographic image will appear in the correct position.

3. Natural interaction
HoloLens main interaction modes are gaze (gaze), voice Command, and gesture (Gesture), which form the basic input elements of HoloLens. At the same time, the traditional keyboard and mouse devices are also supported, the natural way of interaction closer to human habits, improve the efficiency of interaction.

4. General application
The operating system for the HoloLens platform is Windows Holograpic, which is also based on Windows 10 customization. So the Windows ten UWP Universal app can run smoothly on HoloLens. This not only reduces the cost of research and development and migration, but also enables a significant increase in development efficiency. Of course, with a lot of HoloLens's unique features and benefits, there are some issues that need to be addressed in the HoloLens developer version, including narrower horizons, poor gaze experience, weak light disturbances, and weight-endurance issues. But his flaws, HoloLens brings a real mixed reality experience, with a strong sense of impact, the future will be great.

Second, the development environment construction

The following environments need to be prepared before developing HoloLens (the environment I am preparing below)

1. Visual Studio Update 3

2, HoloLens Emulator (Build 10.0.14393.0) (with real machine can not use simulator)

3. Unity HoloLens Technical Preview

4. Install in Windows SDK (10.0.14393.33) vs

More detailed environment configuration can be consulted.

Https://developer.microsoft.com/en-us/windows/holographic/install_the_tools

Third, integrated Holotoolkit-unity project

After creating a standard Unity project, we need to integrate Microsoft's official Holotoolkit-unity project. The Holotoolkit-unity project is Microsoft's official open source project to help developers quickly develop HoloLens applications, enabling them to quickly integrate basic input, spatial mapping, and scene matching features into their projects.

For a detailed introduction to this project, you can refer to Microsoft's Open source project https://github.com/Microsoft/HoloToolkit-Unity

Integration steps

1. Download Holotoolkit-unity Code Https://github.com/Microsoft/HoloToolkit-Unity.git

2. Open Holotoolkit Project with unity, right click on assets, export package

3. Use the Holotoolkit-unity package in your own project

Create your own Unity project, Assets, import package, and Custom package Select the Holotoolket-unity package that you just exported, you can see the Holotoolkit menu after successful import

The following is the structure and content of this project
? Input Directory
? GazeManager.cs for fast integration of gaze ray characteristics;
? GestureManager.cs for fast integration of gesture recognition features;
? KeywordManager.cs for fast integration of voice command features;
? The CursorManager.cs is used for fast integration of visual gaze components.
? Sharing directory
? The sharing Prefab component is used to quickly integrate scene sharing features.
? Spatialmapping Directory
? Surfaceplane Prefab components are used to describe and render real-world surfaces;
? The spatialmapping Prefab component is used to quickly integrate spatial mapping features;
? The remotemapping Prefab component is used to quickly integrate the remote spatial mapping information into the feature;
? Spatialsound Directory
? The UAudioManager.cs is used to quickly integrate spatial sound characteristics.
? Utilities Directory
? Billboard.cs is used to follow the user's line of sight characteristics;
? Tagalong.cs is used to implement the following user movement characteristics;
? The main camera prefab component is used to quickly integrate HoloLens standard master cameras.

Iv. Building a scene

1. After creating a new blank scene, we need to delete the original main camera object, and drag a main camera Prefab component from the Holotoolkit/utilities/prefabs directory into the scene, so that the integration satisfies the HoloLens The basic requirements. The main camera. For HoloLens, the primary camera render background is set to solid color and the color to RGBA (0,0,0,0). Because any pure black color will be HoloLens rendered transparent, to achieve the purpose of not blocking the real world. In addition, HoloLens recommends a camera with a close-up distance of 0.85, which is best for the real human eye experience. At the same time the main camera position must be reset to the world 0 points, namely XYZ (0,0,0), any holographic image will be in this as the origin in the surrounding world to draw out.

2. Click ' Create empty ' to create an empty game object and name it input to add the core Script component for the input object, GazeManager.cs, GestureManager.cs, HandsManager.cs, respectively. and KeywordManager.cs. This integrates the three core features of gaze, gesture, and voice commands. For gaze rays, gesture recognition, and voice command functions, a singleton is recommended for management, which avoids confusion and facilitates management. Also, setting a visual pointer to the gaze can improve the user's interaction experience and efficiency.

3. Drag the Cursorwithfeedback Prefab component into the scene from the holotoolkit/input/prefabs/directory. This way, when staring at a holographic object, a visual gaze component appears on its surface. When the user gesture is recognized, a blue palm image appears, which tells the user to be able to manipulate it.

4. Create a Cube object and create a new C # script named HoloTest.cs.

usingUnityengine; usingSystem.Collections;  Public classHolotest:monobehaviour {//Use this for initialization    voidStart () {}//Update is called once per frame    voidUpdate () {} Public voidOnSelect () {//randomly changing object colorGameobject.getcomponent<meshrenderer> (). Material.color =NewColor (Random.range (0,255)/255f, Random.range (0,255) /255f, Random.range (0,255) /255f); }  }  

5. The function of the HoloTest.cs script is to randomly replace the object's material color, following the preset onselect message name in GestureManager.cs
Enter the Input Component View tab to configure voice commands for the KeywordManager.cs component. The corresponding component behavior will be executed when the voice command is triggered. In this example, when I say "test", the cube's OnSelect method is triggered to randomly change the cube color.

V. Compiling the project

1, in order to meet the needs of HoloLens, we need to open Virtual Realitysupport in Player Settings, and Select Windows Holographicin the drop-down list, This is the only way HoloLens will render this app as a 3D application, which is critical.

2. At the same time, from the toolbar Edit->project settings->quality tab, set the UWP platform default quality to fastest. This is to reduce the performance overhead and the official recommended frame rate is 60fps.

3, Build Settings view to select the target platform for Windows STORE,SDK for Universal 10, the other set to the Unity C # projects Easy debugging, click the Build button to start compiling the UWP project.

Vi. deployment of debugging Applications

Use Visual Studio to open the compiled UWP project, set it to x86(HoloLens system type, non-computer)on the debug (Easy Debugging) option.

After connecting the HoloLens to the PC and completing the Build and Deploy, we open the app in HoloLens. The actual effect is shown. When I use gestures to click Cube, it randomly changes color, and when I speak the Voice command "test", the cube will still change colors normally.

Vii. Summary of development

It's easy to develop HoloLens apps using the Unity engine, and most processes are no different from developing UWP projects. However, there are still a number of areas to be aware of the minefield and special requirements, the following is part of the attention:
1. Main Camera must be configured according to the official requirements, background solid color and RGBA value (0,0,0,0), so as to avoid blocking the real content;
2. Gaze gaze characteristics require us to use the Raycast to achieve, pay attention to processing Ray misses the target situation, the default gaze maximum distance of 15 meters , if the object is not hit, the use of the possibility of a null reference exception;
3. The use of Windows-specific APIs, such as gesture recognition, photography, and voice commands, requires the use of HoloLens-specific APIs for spatial mapping and scene matching;
4. Many other details of the experience, such as visual gaze components, the target area of the visual guidance component, using them to give user tips, can help users understand the application of how to improve the use of experience.

HoloLens Development notes using unity to develop a simple application