Arkit and Arcore analysis, structure, principle introduction

Arkit and ARCore are three parts: camera attitude estimation, ambient perception (plane estimation) and light source perception.
ARCore part of the source code: Https://github.com/google-ar/arcore-unity-sdk/tree/master/Assets/GoogleARCore/SDK;
Arkit Api:https://developer.apple.com/documentation/arkit
ARCore api:https://developers.google.com/ar/reference/ -camera attitude estimation

Motion tracking aspects are vio.
Arkit is the feature point method, sparse point cloud: HTTPS://WWW.YOUTUBE.COM/WATCH?V=RCKNUAYCSJK

Arkit recognizes notable features in the scene image, tracks differences in the positions of those features-across video f Rames, and compares that information with motion sensing data. --https://developer.apple.com/documentation/arkit/about_augmented_reality_and_arkit

ARCore has speculated that the direct method estimates the semi dense point cloud. But Google itself says it's a feature point, and it should be sparse:

ARCore detects visually distinct features in the captured camera image called feature and points s to compute it change in location.
-–https://developers.google.com/ar/discover/concepts -Environmental awareness (planar detection)

Not quite understand the principle, picked some original text.
Arkit:

"... can track and place objects on smaller feature points as ..."
“... Use hit-testing methods ("The Arhittestresult class") to find real-world surfaces corresponding to a point in the camera Image ... "
"Can use hit-test results or detected planes to place or interact with virtual content in your scene."
No vertical plane detected

ARCore:

"ARCore looks for clusters of feature points this appear to lie on common horizontal surfaces ..."
"... can also determine each plane ' s boundary ..."
"... flat surfaces without texture, such as a white desk, may is detected properly ..."
Demo Video: https://www.youtube.com/watch?v=aSKgJEt9l-0 -light source Sensing

Don't know for a while. Arkit

Reference Blog: http://blog.csdn.net/u013263917/article/details/72903174
The IPhone x adds truedepth camera and also supports arkit use. First, Introduction

Arkit Framework

Augmented Reality (mainstream) based on 3D scene (Scenekit) implementation
Augmented reality based on 2D scene (Spriktkit) implementation

The relationship between Arkit and Scenekit

Arkit is not a framework that can be run independently, but must be scenekit together. Arkit realizes the camera captures the real world image and restores the three-dimensional world Scenekit realizes the 3D model of virtual reality in the image

We focus Arkit Second, Arkit

The view that displays the virtual augmented reality of the Arkit in the framework Arscnview inherits from the Scnview in the Scenekit framework, and Scnview inherits from Uikit in the UIView framework.
In a complete virtual augmented reality experience, the Arkit framework is responsible for transforming the real world picture into a 3D scene, which is divided into two parts: the Arcamera is responsible for capturing the camera screen, and the arsession is responsible for setting up 3D scenes.
There is no direct relationship between Arscnview and Arcamera, which is a kind of arsession to build bridges through AR conversation, which is a class of arkit in the frame.

To run a arsession session, you have to specify an object called a session-tracking configuration: arsessionconfiguration, The main purpose of the arsessionconfiguration is to track the position of the camera in the 3D world and to capture (such as plane capture) a number of feature scenes, a class that is relatively simple but powerful.

Arsessionconfiguration is a parent class, and in order to better see the effect of augmented reality, Apple has officially suggested that we use its subclass arworldtrackingsessionconfiguration, which only supports the A9-chip model. Which is the model 2.1 after iphone6s . Arworldtrackingsessionconfiguration and Arframe The main participants of Arsession Bridge are two arworldtrackingsessionconfiguration and arframe.

The Arworldtrackingsessionconfiguration (session-tracking configuration) is to track the direction and location of the device, and to detect the real-world surface of the device's camera. It internally implements a series of very large algorithmic calculations and calls the necessary sensors on your iphone to detect mobile phone movement and rotation or even tumbling.

Arworldtrackingsessionconfiguration inside is the vio system.
The arworldtrackingsessionconfiguration mentioned in this article have been discarded in the latest iOS beta8, so the following is changed to Arworldtrackingconfiguration

When Arworldtrackingsessionconfiguration calculates the position of the camera in the 3D world, it does not hold the position data itself, Instead, the computed position data is given to Arsession to manage (echoing the previous session management memory), and the corresponding class of the camera's position data is Arframe
Arsession Class A property called Currentframe, Maintenance is arframe This object Arcamera only responsible for capturing images, do not participate in the processing of data. It is part of the 3D scene, each 3D scene will have a camera, it feels that we look at the vision of the object.
2.2. Arsession

There are two main ways to get camera location data arsession

First: Push. Real-time access to the camera location, by Arsession initiative to inform users. By implementing Arsession's proxy-(void) session: (Arsession) session Didupdateframe: (Arframe) frame to obtain
The second type: Pull. When users want to, take the initiative to get. Arsession the property currentframe to get 2.3. Arkit Work Complete process arscnview load scene scnscene scnscene boot camera arcamera start capturing scene capture scene Arscnview start to hand over scene data to session Session through the management Arsessionconfiguration implementation of the scene tracking and return a arframe to Arscnview scene Add a child node (3D object model)

The significance of capturing the camera 3D position is to be able to calculate the true matrix position of the 3D object model relative to the camera when adding the 3D object Model arsessionconfiguration.

Third, the API analysis

-arorientationtrackingconfiguration
Tracks the device ' s movement with three degrees of Freedom (3DOF): Specifically, the three rotation; only three tracks, not as follows.
-[arworldtrackingconfiguration] (https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration)
Responsible for tracking camera and detecting plane. Tracks the device ' s movement with six Degrees of freedom (6DOF).
The main content of completing slam work should be in this.
But to start one of the simplest AR, just need:

Let configuration = Arworldtrackingconfiguration ()
configuration.planedetection =. Horizontal
SceneView.session.run (Configuration)

The specific implementation is encapsulated.
Then with the important arsession class and the Arscnview interface:
Due to many interfaces, only the column section, detailed to the official website or reference Bowen: http://blog.csdn.net/u013263917/article/category/6959089
Apple has a feature of its own, face-based AR experience. You can use truedepth camera to track facial expressions, pose, topology, and so on. World based AR experience corresponds to other vendors ' SDKs, such as Arcore.
For example, some common information interface:
The position of the virtual object, Arframe–>aranchor in the transform
The position of the camera, the Arframe–>arcamera displaytransform contains the pose
View matrices, projection matrices: Projectmatrix,viewmatrix in Arframe–>arcamera
-arsession

Tables	cols	cols
Configuring and Running a session	Func run (arconfiguration, options:ARSession.RunOptions =)	Starts AR processing for the "session" with the specified configuration and options.
	var configuration:arconfiguration	An object so defines motion and scene tracking behaviors for the session.
Responding to AR Updates	var delegate:arsessiondelegate	An object for you provide to receive captured video images and tracking information, or to respond to changes into session Statu S.
	Protocol Arsessiondelegate	Methods can implement to receive the captured video frame images and tracking the from a AR session.
	Displaying and interacting with AR Content	var currentframe:arframe
	Func Add (Anchor:aranchor)	Adds the specified anchor to is tracked by the session.

-arscnview
Arscnview is the interface that the user sees.

Tables	cols	cols
	var session:arsession	The AR session is manages motion tracking and camera image processing for the view ' s contents.
	var scene:scnscene	The scenekit scene to is displayed in the view.
Responding to AR Updates	Protocol Arscnviewdelegate	Methods you can implement to mediate the automatic synchronization of Scenekit content with a AR session.
	Func hittest (Cgpoint, Types:ARHitTestResult.ResultType)	Searches for Real-world objects or AR anchors in the captured of camera image corresponding to a point in the Scenekit view.

-arcamera
There are a number of related topics in the Arcamera class:

Tables	cols	cols
Handling Tracking Status	Trackingstate	The general quality of position tracking available the camera captured a frame.
	Artrackingstate	Possible values for position tracking quality.
Examining Imaging Parameters	Imageresolution	The width and height, in pixels, the captured camera image.
Applying Camera Geometry	ProjectionMatrix	A transform matrix appropriate for rendering 3D content to match the image captured by the camera.
	Projectionmatrixfororientation:	Returns a transform matrix appropriate for rendering 3D content to match the image captured by the camera, using the Speci fied parameters.
	Viewmatrixfororientation:	Returns a transform matrix for converting from world spaces to camera spaces.

-arframe
The topics in the Arframe class can see the slam input-output interface

Tables	cols	cols
Accessing captured video Frames	Capturedimage	A pixel buffer containing the image captured by the camera.
	Captureddepthdata	The depth map, if any, and the captured along with the video frame.
Examining Scene Parameters	Camera	Information about the camera position, orientation, and imaging parameters to used the frame.
	Lightestimate	An estimate of lighting conditions based on the camera image.
	Displaytransformfororientation:	Returns a affine transform for converting between normalized image coordinates and a coordinate spaces appropriate for Ren Dering the camera image onscreen.
Tracking and finding Objects	Anchors	The list of anchors representing positions tracked or objects detected in the scene.
	Hittest:types:	Searches for Real-world objects or AR anchors in the captured image.
Debugging Scene Detection	Rawfeaturepoints	The current intermediate results of the scene analysis Arkit uses to perform world tracking.
	Arpointcloud	A collection of points in the "the" AR session.

The number and identitiers of feature points can be seen through arpointcloud of feature points.
What kind of feature points are used. You may need to look at information such as the dimensions of the identitiers. The descriptor of SIFT characteristics is 128 d.

-arlightestimate
Slightly ARCore

Google has released a arcore for Android studio, Unity, Unreal, and the web environment.
API official website: https://developers.google.com/ar/reference/.
SDK on Github:https://github.com/google-ar;
2017.12.15 update: Google added Java, C platform support. (The original Android studio part was in Java)
The structure of Arcore is more flexible,
In AR, a virtual object is drawn in a real scene, and the information that the developer must know before it is drawn:

Position posture of virtual object
View matrices, projection matrices
Illumination information of Virtual objects

All of this information can be obtained from Arcore. Let's look at the sample in the Java platform provided by Google first.
First of all, the drawing logic in sample, about the class of drawing:
Backgroundrenderer is used to draw data captured by the camera.
Virtualobject is used to draw Android small robots.
Virtualobjectshadow is used to draw shadows for Android robots.
Planerenderer is used to draw the plane that the SDK recognizes.
Pointcloud is used to draw the point cloud identified by the SDK.
Using OpenGL ES Drawing: Configure Glsurfaceview, implement Glsurfaceview.renderer interface. Parts of the drawing can be replaced with some more modern 3D graphics frames.

Mention the renderer interface;
' Public interface Renderer {
void onsurfacecreated (GL10 gl, eglconfig config);
void Onsurfacechanged (GL10 gl, int width, int height);
void Ondrawframe (GL10 gl);
}
Onsurfacecreated This method is invoked when a surface can be created or recreated. In this callback, you can do some initialization things. Note that this method runs in an OpenGL thread with an OpenGL context, so you can perform an OpenGL call here.
Onsurfacechanged This method is invoked when the surface of the drawing can be changed. The outside may change the size of the control, so we need to update our viewport information in this call so that we can draw it exactly to the screen.
void Ondrawframe Core method. Called at the time of drawing. Each time it is drawn, it is invoked once, that is, each frame is triggered once. Here is the main drawing logic.
Reference: HTTPS://JUEJIN.IM/POST/59AC1F2BF265DA249517AC72
See the specific implementation of sample bar.

To obtain the Pose,viewmatrix and Projectmatrix of virtual objects and the specific interface for lighting these necessary information:
Let's look at the Java platform first. Https://developers.google.com/ar/reference/java/com/google/ar/core/package-summary the pose of virtual objects:

Arcore stipulates that when you want to place a virtual object, you need to define an anchor to ensure that the Arcore tracks the position of the object over time.
Anchor class

Getpose can get the current position of anchor. Sample called Anchor.getpose (). Tomatrix (Manchormatrix, 0); got the anchor position matrix, a 4x4 model-to-world transformation matrix, stored in Column-major order. view matrices and projection matrices:

The camera class stores this information:

The camera property will update with the Session.update () is called. So let's see how the session class updates camera.

Session class, the session class manages the state of the AR system, and the session is the main entry point to ARCore API.

The last update () updates the state of the Arcore system, including getting a new camera frame, updating the location of the device, updating the anchor location, and updating the detected plane. The new camera property is obtained by the Getcamera () of the frame.
Frame class

HitTest is Google's hit test interface, which detects whether a user clicks on a plane or loads a virtual object.

The light does not care for the moment.

The C platform is similar to java.
Session class:

Frame class, including Frame,camera,hitresult in Java, lighting and other classes:

Anchor is included in the Trackable class: