UWP hand-painted video creation tool "To Draw Pro" technology sharing series, uwp to draw

UWP hand-painted video creation tool "To Draw Pro" technology sharing series, uwp to draw

Let's start with the story I drew and the original intention of writing this article.

At the beginning of this year, I was still in Beijing, working on the Windows and Android terminals of Face recognition technology in Face ++, and AI would eventually realize all the beautiful dreams of the world. At this time, I have been away from UWP, even C # For a long time, and wrote C ++ and Java. At that time, I did not really think that I would go back to the UWP camp again, until April, A very occasional opportunity, one of my Microsoft friends and an old friend of Face ++ found me and asked me about my work in Face ++, I thought it was just a simple conversation after I had not been in touch for a long time until he mentioned: in Shenzhen, there was a startup company called "laihua", which made hand-drawn video tools and platforms, ask me if I want to join them. Frankly speaking, at that time, I only heard of video platforms, but I didn't even know what it meant. However, I was surprised that the company launched UWP apps, but there were no iOS and Android apps at all, and they were Microsoft's key partners, i'm also very interested.

You can get a basic understanding of hand-drawn videos and the creation process through the official website and UWP. A hand-drawn video is a video representation of various texts, images, sounds, music, or animations that are drawn or moved with a real hand or pen when a video appears. In the Web tools and UWP for painting, the creation process is very simple. You can select or upload svg, input some text, select background music, and set the animation duration and mode as needed, A few minutes of hand-drawn video is ready.

I like this kind of video creation and presentation. I am also very interested in the company's attitude towards UWP and the technical difficulties of this product. So I accepted the interview invitation from the company, come to Shenzhen from Beijing for an interview. The company kindly helped me book a round-trip ticket and accommodation, and even arranged a connection to and from the airport. This made me start the interview with a good first impression. The interview was arranged on the weekend, CEO Wei introduced many hand-drawn video application scenarios and the present situation and development goals of the platform/UWP, which made me more interested in this video creation form, in particular, we introduced the application of hand-drawn videos for weddings. Many memories are difficult to photograph, and many feelings and ideas are difficult to photograph, hand-drawn videos can be completed through painting, text and music. This really touched me. For the first time, I decided to join the painting and put it into the tide of hand-drawn videos. So I decided to return to Beijing and leave Face ++ and come to Shenzhen to draw a story. This is the beginning of my story.

The above describes the process of painting and the UWP App, which is the focus of this article.

At the beginning of the painting, I read through the UWP App code and found that there are indeed many advanced technical points, but also many problems, such as scalability, coupling, performance, and functions. So we decided to reconstruct this product from the ground up and create a brand new "come to Draw Pro". After three months of the world, this product was finally officially launched. (Make a small advertisement and search for "paint Pro" in the Windows app store. Currently, the company is positioned as a paid app, but there is a one-month trial period. If you are interested, you can ask me for free code. Here is the IT House Report: https://www.ithome.com/html/win10/324042.htm ). Based on the implementation of all the tools of the old version of "laifeng", we have implemented more Ink paint and Dial functions, better SVG parsing and drawing, and better text painting processes, insert video and gif images. The original intention of writing this article and the next series of articles is to give a comprehensive review of the technologies and problems encountered in the development of "come to Draw Pro, we hope to share this with more people interested in UWP and hand-drawn videos.

 

Now let's get down to the point and start to share the implementation methods and effects of the functions of "come to Draw Pro". A series of articles will be provided to describe each function point in detail.

1. Win2D-Basics of hand-drawn video Rendering

Win2D is a 2D graphics GPU rendering SDK Based on Windows Runtime. It can be used in UWP, Windows Phone, and Windows Runtime apps. The programming languages can be C ++, C #, or VB, I believe that students who have tried graphic rendering in UWP are familiar with Win2D. After all, I recommend the implementation method. GitHub address: Please Sample for your reference, in Windows Store can also download: https://www.microsoft.com/zh-cn/store/p/win2d-example-gallery/9nblgggxwt9f? Rtc = 1, the following is Sample:

2. parsing and drawing SVG

As you know, SVG is a vector graph format. Unlike bitmap, it is composed of XML, and node information includes path, stroke, and fill, it represents the path, path border, and filling rule of SVG respectively. Our operations on SVG are first SVG parsing and static display, and then based on the original path composition and sequence of SVG, combined with the sequence of hand-drawn habits, the total length of the path and the total length of the drawing are determined in each frame which part of the path is displayed or which part of the shape is filled. Let's take a look at the Implementation Effect in the application. The following three figures show an SVG Painting Process:

3. Drawing PNG, JPEG, BMP, and other bitmaps

PNG, JPEG, BMP, and other image formats have different compression and encoding formats. However, because they all belong to bitmaps, the components are pixels, an array of width * height length is used to store the pixel values of each pixel. The pixel values may consist of multiple formats, such as ARGB, BGR, and YUV. Because bitmap has no path, we cannot draw the creation process like SVG by default, so we chose a relatively simple rendering method: draw from one corner of the image to another, such as the upper left to the lower right corner, to maintain a stable speed and direction, this is the process of drawing.

Of course, the above is just the simplest and most crude processing method. In actual scenarios, many SVG not only contain paths, but also bitmaps. For example, process an image in PS, import it to AI, manually draw some paths, and finally export SVG. The correct SVG painting process should be based on the path in AI and the Stroke of the corresponding position in the bitmap. In processing bitmap, we can also add ways for users to outline paths and enrich the presentation methods. You can also perform automatic edge profiling and image cutting operations on bitmap, which will be shared in detail later.

 

4. Text parsing and drawing

Text is an important display and expression method in hand-drawn videos, and it is also a relatively complex process of parsing and drawing. After a font is determined, the corresponding text will be taken out of the TTF file of the corresponding font for display, and the path data representing all the edges of the text can also be obtained. However, in hand-drawn videos, the display mode is to depict the filling of the font, rather than the edge. In addition, the edge of the font itself does not have any order or regularity, but simply forms the edge of the font, therefore, we need to process the edge path as follows:

Obtain edge path data-> perform horizontal or vertical rays on points in the path point by point, record all intersection point sets-> edge path point set and Intersection Set to calculate the Intermediate Point Set-> sort the normal text of the intermediate point set in top/bottom/left to right-> sort by point distance regroup-> draw the path based on the processed point set.

5. drawing Ink and using Surface Pen and Surface Dial

Ink handwriting is widely used in Windows 10, such as plotting, writing, and hand-drawn videos. Microsoft's Windows SDK provides great InkToolbar and InkCanvas, which can be displayed with several paint brushes by default, such as ballpoint pen, pen, pencil, and highlighter. What we need to do in hand-drawn videos is to extend the static display supported by the SDK by default to support dynamic rendering and export the video support. The Ink tip, such as a ballpoint pen, can be taken out and used directly. The Ink effect of the pencil, the stroke width of the pen, and the color mixing effect of the highlighter are as follows, you need to perform separate processing on your own. We will explain in detail the processing of each paint brush later.

InkToolbar and InkCanvas support Surface Dial and Surface Pen operations by default. Our applications also provide more extended support for Surface Dial, such as color selection and paint brush selection. The following is an operation diagram of the InkToolbar and InkCanvas and the Surface Dial used in the application (from the IT house ):

 

6. Insert a video file into a hand-drawn video

You can insert videos into hand-drawn videos to enrich the representation. In common scenarios, you can add hand-drawn character materials next to a video to enable adult comments and narration. Alternatively, you can add pasters or items to a video to dynamically change content next to a video or in a video, in this way, it is richer than simply processing videos and adding stickers, which can often be seen in TV programs, such as inserting videos into hand-painted videos and turning them into a dynamic background. One of the following scenarios:

The MediaElementPlayer player of the system is used for processing in the program during operations and previewing. When exporting a video, you must independently process the inserted video frame and audio track.

 

7. video generation and export

After the above operations are completed, we need to export it into a video file, such as MP4 and WMV. This generation process mainly includes two parts: ① It is to use Win2D for background rendering operations to restore the real rendering process and speed, saves the bitmap of each frame according to the specified frame rate. ② uses a method similar to FFMpeg to save the frame sequence as a video according to the specified frame rate, simultaneously process overlapping background music and inserted audio tracks to complete video file generation.

 

By now, the basic process of parsing, rendering, operating, and exporting the main elements of hand-drawn videos has been completed. Today's article will give a general introduction to the implementation principles of each part. Later, we will give a detailed explanation of each part, especially in terms of SVG and text drawing, as well as detailed principles and implementations, and various problems encountered in the development process, or their own, or Microsoft Systems. And various paint brush implementation methods.

Thank you!