Research and development summary of virtual camera under MacOS Platform

First, Background introduction

Virtual camera, as the name implies, is the use of software technology to virtual a camera hardware devices for users. When we need to process and re-export the video image, the virtual camera has a lot of value. The information on how to implement a virtual device on Windwos is very rich, and there are a lot of help documents in the Windows Driver kit. This blog post mainly summarizes some of the experiences of developing virtual cameras under Mac. In fact, there are many virtual camera products under Mac, such as Camtwist, Cammask, Camwiz, ManyCam and so on. But the information on how to develop a virtual camera device under a Mac is extremely scarce. After a search, a keyword was found: Coremediaio. It is understood that Coremediaio is a framework under the Mac that is used primarily for processing video images. The Coremediaio framework has a Device abstraction layer (DAL) that resembles the hardware abstraction layer (HAL) CoreAudio under the Mac. The HAL is primarily used to process audio streams sent by audio hardware, while the DAL is used to process video streams of video devices. Therefore, using the DAL plug-in framework, you can simulate a camera device for use by upper-level users.

The Coremediaio dal has a sample project that simulates a device named "Sample" that delivers video frame delivery through the simulation data provided by the underlying Kext module. For this demo to be really used, there are a few things to be aware of. The following command is recommended for installing pre-compiled programs in the Readme file in the demo:

Debugsudo darwinup Install {path to Coremediaio folder}/prebuilts/sample-debug.tar.gz//releasesudo darwinup Install { Path to Coremediaio folder}/prebuitls/sample-release.tar.gz

After this installation, the virtual camera cannot be found immediately. On the one hand, because kext provides more than 800MB of analog data, it takes some time to load into memory, and on the other hand, the Kext module in Prebuilts is unsigned. And OSX since Mavericks began to require the Kext module must be signed, the system will automatically load. Otherwise, you need to shut down system Integirty Protection (SIP) and manually load the Kext module for the demo to work properly.

Second, how to compile the project

Before we start developing custom virtual cameras, the first step is to figure out the organizational structure of the demo project. Two documents are included in the demo project, respectively, explaining the project structure and working principle of the Dal plug-in. These two documents should have a good look before they are developed, and will be of great help in understanding the entire code structure and working mechanism. In addition, how to compile the entire demo project is also a big problem. Because the CoreAudio module is missing from the downloaded project, it is necessary to manually download the CoreAudio module to join the project. Then there may be some syntax errors that need to be modified, depending on the system version and the version of Xcode.

1. Add the CoreAudio module. The default project does not contain the CoreAudio module, so direct compilation will have many link errors:

: Core Audio Utility Classes. (may require an Apple ID login). Download and add the entire folder to the demo project to compile. if compile or error , you can compile the CoreAudio module separately, and then add the CoreAudio header file and the static library in the demo project to compile, it should be able to solve the problem of the compilation.

2. syntax modification. Perhaps because of the version of the MacOS SDK, there are some syntax errors that need to be modified during compilation, as follows:

The workaround is also simple:

There are some assignment problems, but basically the problem of compiling the standard, it is not difficult to solve.

Third, plug-in module modification

1. resolution. The official Demo project offers several resolutions: 720x480, 1280x720, 1920x1080. If you want to increase your resolution, you can change these resolutions directly.

In this case, the width of the resolution is modified, and note that the frame rate is also modified here. Note the color mode used in the demo project three resolutions is UVVY422 Oh! So if you want to use the color mode of the demo project, you need to convert the image to UYVY422 format.

2. color mode. How do I change the color mode? There are a total of two places to be modified, in addition to the above KCMVIDEOCODECTYPE_422YPCBCR8 needs to be modified, in the cmio_dpa_sample_server_stream.cpp there is a place to modify:

This specifies the color mode supported by the plugin. So, what color models does plugin support altogether? Look at the enumeration members in the enum to find out:

3. Basic equipment information. Equipment basic information is mainly the name of the device, convenient for users to identify and select the use. There is only one place that needs to be modified:

4. The Create server failed issue. In the plugin entry function, there is a snippet of code:

The developer explained in detail in the comments that the purpose of debugging here is to try to start assistant manually. In the actual test, bootstrap_create_server () often fails to throw an exception, causing the Ingress function to end prematurely, thus creating a virtual device failure. The note also explains that the plist file is typically used to create the Assistant service when the system starts. This way, bootstrap_loop_up () skips the manual creation of the Assistant service after it finds the assistant service.

In this view, the Plist file located in/library/launchdaemons is essential. The plist file under/library/launchdaemons/is actually specifying which programs start from running with the system. The contents are as follows:

<?xml version= "1.0" encoding= "UTF-8"? ><! DOCTYPE plist Public "-//apple//dtd plist 1.0//en" "Http://www.apple.com/DTDs/PropertyList-1.0.dtd" ><plist Version= "1.0" ><dict><key>label</key><string>com.apple.cmio.dpa.sample</string ><key>programarguments</key><array><string>/library/coremediaio/plug-ins/dal/ Sample.plugin/contents/resources/sampleassistant</string></array><key>machservices</key ><dict><key>com.apple.cmio.dpa.sample</key><true/></dict></dict></ Plist>

Iv. Kernel Module Modification

1. Remove the analog frame data from the demo. The demo project provides three-resolution analog data with 30 frames of data per resolution. The three-resolution data add up to more than 800 trillion, it is necessary to remove them when customizing. According to the demo project configuration, the analog frame data is compiled into the Kext module as section data:

So the way to get rid of the analog data here is very simple, the "otherLinker Flags" inside the contents of all deleted. Of course, the parts of the code that use these section data are also deleted.

2. kernel signature. OSX has introduced a signature mechanism to kext development since Mavericks . All unsigned kext module systems are no longer automatically loaded. Therefore, to allow the system to automatically load the third-party developed Kext module, the developer needs to apply to Apple to sign the Kext certificate (see here). A generic developer certificate cannot be properly identified by the system, even if it is normally signed. After signing, the Kext module can be signed for verification:

In the absence of a certificate capable of signing Kext, SIP can be switched off into test mode. This allows the kext to be loaded manually even if it is unsigned, making it easy to test the program.

3. the Kext module is loaded automatically. Here's a little bit odd : The signed kext module is automatically loaded by the system when the system restarts, but the Kext module service is not accessible through plugin. You must call Kextunload once and then kextload once to function. It's not clear what this is for the moment, but it's also a problem, depending on the solution of other virtual camera products. For example, the Camwiz solution is:

(1) Write a plist file under the/library/launchdaemons/file with the following contents:

<?xml version= "1.0" encoding= "UTF-8"? ><! DOCTYPE plist Public "-//apple//dtd plist 1.0//en" "Http://www.apple.com/DTDs/PropertyList-1.0.dtd" ><plist Version= "1.0" ><dict><key>Disabled</key><false/><key>Label</key>< string>com.apple.vcam.driverreloader</string><key>programarguments</key><array>< string>/library/coremediaio/plug-ins/dal/sample.plugin/contents/resources/reloadkext.sh</string></ Array><key>runatload</key><true/></dict></plist>

(2) Write a shell script reloadkext.sh, and package it into the resources folder of Sample.plugin, with the following contents:

#!/bin/sh/sbin/kextunload "/library/extensions/iovideosample.kext"/sbin/kextload "   /Library/Extensions/ Iovideosample.kext "

This will call the script every time the system restarts, unload the Kext module and load it again. This solves the problem of the Kext module. In fact, it is best to remove the Kext module without the need for a hardware layer. But the entire demo project code complex, the document is extremely scarce, want to peel off the Kext module difficult. But there are still a lot of products that have achieved this, such as Camtwist, Cammask and ManyCam. Camtwist more awesome is that in a plug-in virtual out of two devices. One is the YUV color mode, the other is the Bgra color mode.

V. Other MATTERS

1. Program packaging and installation path. According to functional requirements, the entire project installation program is divided into three parts:

(1) *.plugin installation to/library/coremediaio/plug-ins/dal/.

(2) *.kext installation to/library/extensions/. Apple's Official Rules: third-party development of the Kext module can only be placed here, and Apple's own kext module is placed in the/system/library/extensions/.

(3) *.plist installation to/library/launchdaemons/. The plist files that are placed here are all designed to boot up. So when making the installation package, remember to ask the user to restart the system when the installation is complete.

2. file permissions issues. It is best to modify owner permissions for all files in the installation package, otherwise it may not work:

$ sudo chown-r root:wheel *

3. Related commands

(1)Otool with install_name_tool

(2) kext* command (kextload, kextunload, kextstat, kextutil)

4. Install Package Packager or recommend the use of packages, which is a conscience tool. Simple to use, beautiful interface, powerful and practical, is really the choice of making pkg files.

Vi. Reference Links

1. https://developer.apple.com/library/content/documentation/Security/Conceptual/System_Integrity_ Protection_guide/kernelextensions/kernelextensions.html

2. Https://superuser.com/questions/47233/how-can-i-force-a-mac-os-x-kext-to-load-prior-to-login

3. Http://apple.stackexchange.com/questions/163059/how-can-i-disable-kext-signing-in-mac-os-x-10-10-yosemite

4. Https://developer.apple.com/library/content/samplecode/CoreMediaIO/Listings/Prebuilts_Prebuilts_ Readme_txt.html

5. https://forums.developer.apple.com/thread/18019

6. https://macwish.com/kext-signing-for-mac-yosemite/

7. https://developer.apple.com/library/content/documentation/Security/Conceptual/System_Integrity_ Protection_guide/configuringsystemintegrityprotection/configuringsystemintegrityprotection.html

Research and development summary of virtual camera under MacOS Platform