6 years double 11, DataV data visualization

Since 2012, Alibaba's annual Double 11 Singles Day Promotions has launched a large screen that displays transactions in real time in a variety of lively presentations. Real-time data large screen, it is characterized by a variety of large: large screen, large amount of data, display information, can be said to be the holy grail of data visualization.

Alibaba Cloud DataV team has participated in the construction of double 11 Singles Day media digital screen for 6 consecutive years. This year's "China Regional Economic Lightning Map" media touch screen has become one of the core focuses of the double 11 party scene. The order flocked to the seller along the real road, and the effect was shocking. The touch screen allows users to drill down to query data, greatly enhancing data readability. On that day, more than 500 Chinese and foreign media used this as a live data interpretation platform to achieve the perfect integration of new and old media.

This year's double eleven data big screen project, for the first time integrated application "DataV data large screen computing and rendering load balancing", "global logistics trunk visualization model", "DataV MScreen multi-view fusion", "WebGL late visual effects enhancement" and many more DataV self-developed data visualization technology, in the super 12K resolution single-screen 3D rendering, 30+ real-time data source concurrent access scene, still can guarantee silky smooth and shocking visual effects.

In 2016, the double 11 Singles Day media screen, the main vision is Shenzhen 3D city real scene, the building color represents the size of the order amount in the building, the red color indicates the higher the order amount. The panel on the right shows information such as the user's portrait of the crowd.

As a behind-the-scenes player of the big 11 Singles Day data big screen, DataV has tried almost all the paths of technology precipitation after going through several large projects: open source code, productization, packaging solutions, platformization... application scenarios also from double 11 Singles Day e-commerce operations Expanded to smart cities, smart transportation and many other fields. Their journey and thinking along the way are worthy of reference for reference.

The birth of DataV is accompanied by the Alibaba Group's exploration of data-driven operations.

In 2011, Taobao hatched the first data product, Data Cube, which aims to reduce the cost of understanding data by small and medium-sized sellers on the platform through vivid data charts, and then combine the functions of chart interaction to let sellers borrow in actual operations. Store data. The emergence of Data Cube has popularized the value of data operation for both internal and external. At that time, it was the current DataV team responsible for the visual development of Data Cube. They quickly assumed the technical upgrade of Alibaba's internal data large screen. It also promoted the construction of Alibaba's real-time data indicator system.

In 2012, the Double 11 Singles Day had its own media center for the first time – a temporary command center transformed into a training classroom. The hardware of the large screen was supported by two projectors and two notebooks. Such a device looks like today. Very simple and rude. In the past, because the transaction amount was estimated to be too low, it also caused an oolong. The 12-year double 11 Singles Day did not start long ago, and the actual transaction amount exceeded the preset maximum value. The GMV real-time line chart exploded. Although there was no special combing of indicators and lack of experience in visual chart development at the time, the double 11 Singles Day big screens that debuted at the time gave people a huge visual impact and a real sense of data.

Until this year's double 11 Singles Day, the DataV team was responsible for the visual development of the double 11 Singles Day media data large screen for 6 years, the large screen resolution from 1024 * 768 to 12K, the screen area is almost 400 square meters, showing the complexity of the data and the real-time data link The road has changed dramatically.

On the one hand, the technical capability upgrade, the other side of the DataV team positioning also switched from the role of the in-service to the provider of the visualization products and solutions, and the ability to visualize its real-time data is being exported. At the same time, DataV is also transforming from simple data visualization to data visualization analysis.

At present, DataV is fully cooperating with Alibaba Cloud's ET City Brain Project to implement a city-level multi-dimensional visual analysis platform. The first phase of the data command big screen has become a part of the daily work of Hangzhou traffic police, and has also combined with machine intelligence image recognition. The AI scenario begins a new round of technical solution upgrades.

In addition, DataV is also working with many government companies that are embarking on technology transformation to upgrade the data screen to help them quickly find the data in the initial stage of data operation management.

From open source to productization

Early DataV precipitated the datav.js component library and made it open source, but later, DataV abandoned the open source project and provided its productization. Why?

The original intention of open source is to give visualization technology to more data presentation scenarios, but in the promotion process, it encounters a large number of customized issues. Developers encounter a lot of problems in processing data and chart docking (such as D3 rich The demo library is still slightly higher for data developers, so open source has not achieved the original vision. In addition, the datav.js component library architecture also considered support for the lower version of IE browser, but the demand in this area became weaker, so the library was slowly abandoned.

In the process, the DataV team observed that providing a visual component library is not the same as opening a full link for data kanban production. For those who directly face data or want to interact with the data, the component library lowers some of the thresholds. But beyond that, high-performance data visualization involves multiple skills such as style design, data connectivity, and component architecture design. These products are more complete based on a single product. So the DataV team began to ponder the path of productization.

First of all, data visualization is a front-end post in the large company system, but it is a comprehensive manifestation of business understanding, data understanding and visual expression. It needs business, data, front-end and design to complete together. From the pure technology point of view, the front-end components are only a part, how to process the data, how to dock is actually a more crucial step in the visualization of the business. Compared with the familiar development path of ECharts, DataV hopes to connect data, business, front-end and design together in a product-oriented way to truly lower the threshold.

In this way, data development can easily achieve professional chart front-end work, designers and businesses can easily make "moving" data billboards, greatly reducing the cost of engineering and communication between visual multi-stacks.

Secondly, a commercial product, through feedback with customers, developers can more accurately understand the real needs of users. Although the open source front-end components are widely used, developers themselves do not know what kind of scenes others use, it is difficult to touch the pain points and demands. From the front-end point of view, after the development of the basic chart is completed, it will also encounter bottlenecks in the direction.

DataV's current positioning is also a tool for visualizing and analyzing on the big screen.

In terms of function, the most important thing is to provide a drag-and-drop large-screen interface configuration tool, as well as a variety of templates, such as the PPT template can easily achieve the effect of double 11 large screen. It also includes a variety of data connectivity capabilities, including common database types, API interfaces, and more.

In terms of features, in addition to the conventional cool basic chart components, DataV has made great efforts in the field of WebGIS, integrating a variety of WebGIS capabilities. Very good for d3.js, leaflet.js, mapbox.js, native canvas, SVG, traditional back-end drawing map tiles, mapping of various geographic data, projection, and event response of multiple layers of maps. Integration. General BI or visualization tools are easier to implement on the underlying chart, but the GIS part is much weaker. DataV makes it easy to integrate multi-layer multi-drawing technology and geo-related data integration through an interfaced configuration.

In 2016, DataV front-end development engineer Mill shared “Development of Double 11 Singles Day Data Large Screen Technology”. Taking the example of Double 11 as an example, it introduced the process of DataV from the initial exe program to large-scale WebGL application.

DataV.gl: WebGL-based smart city visual analytics solution

After years of research and practice on WebGL, DataV introduced the DataV.gl component to improve rendering performance on a large scale and customize 3D digital cities. From this point of view, the world has already achieved the leading level.

DataV packages 3D visualization capabilities into an easy-to-use component. Users can import 3D model files and edit data and styles through graphical menus like other charts. DataV.gl supports 100,000. The level of building rendering supports the presentation of pipelines and natural landscapes within the city. Users can also customize the lens trajectory and realize the free exploration of urban images.

From a technical perspective, DataV.gl greatly improves the computational efficiency and engineering efficiency of 3D urban geographic visualization. Using browser-based classic GPGPU technology, using the computing power of the graphics card, DataV.gl supports massive data smooth rendering of computationally intensive visualization schemes such as flow fields and force maps. Compared to the traditional CPU-based Canvas force map rendering solution, DataV.gl's GPU-accelerated power map rendering solution is more than 30 times better.

At the same time, DataV.gl supports urban data planning files (CAD and other formats), standard vector geographic data as input data, and through the geographic data visualization mapping service, directly to the 3D urban building model, which is more realistic than traditional designers using 3D software modeling. Compared with several weeks, the efficiency has increased dramatically.

In the past two years, Uber has made great achievements in the field of 3D visualization of geographic information, and the released deck.gl front-end library has become the industry benchmark. However, a single source visualization technology framework cannot provide a big data visualization solution for multiple scenarios across industries. At the same time, deck.gl has many limitations in meeting individual needs and front-end technology stack selection.

Uber deck.gl is a visual front-end library that can solve the needs of data visualization development. Other links require users to find solutions themselves. Datav.gl is a set of high-performance visualization solutions for massive data. Users can "one-stop". Complete the data visualization business.

If you use a car analogy, Uber deck.gl is like a car engine, datav.gl is a complete vehicle; only buy the engine can not get on the road, you need other parts to assemble.

In addition to this, the differences between datav.gl and deck.gl are:

(1) From the functional positioning, datav.gl is more widely covered than deck.gl, not only provides a visual library function, but provides a complete set of big data visualization solutions.

(2) Differentiating from the target user group, deck.gl is biased towards the developer group, and datav.gl is biased toward enterprises and government users.

(3) From the comparison of user demand adaptation, deck.gl and React are deeply bound, and the basic official visualization component is convenient to use in front-end development, but some individual requirements need to deeply analyze the deck and React framework to develop, relative cost. Higher. Datav.gl provides a developer version that allows users to develop custom components using either official components or their own technology stack and development habits.

(4) From the perspective of rendering efficiency, datav.gl and deck.gl are both based on WebGL, and the basic performance is similar. Datav.gl has advantages in some specific massive data visualization scenarios, as described above.

(5) From the aspect of visual material resource integration, datav.gl accesses some resources such as live streaming media and 3D modeling, and provides fool-like components; deck.gl is relatively complicated.

The future of DataV

Regarding the future direction, DataV's focus is mainly on visual ecology and technology exploration.

Although the data big screen can serve all walks of life and the market potential is huge, but from the various aspects of the big screen landing, or from the field knowledge required to finally cut into the industry, this market needs more ecological partners to join, play out Not the same spark. For example, in the ET city brain project, many ecological forces and DataV work together to form a complete set of urban-level data visual analysis large screen solutions.

Of course, as a technical team, DataV also places great emphasis on visual developers on the platform. For this reason, they have released a developer version that allows developers to perform secondary customization development. By following certain development specifications, they can integrate homemade components. Products can be graphically configured like native components. As well as supporting developers to publish their own components to the Alibaba Cloud market, the ability to visualize is realized – this is the continuation of the initial efforts of the DataV team to promote the development of data visualization and to find a commercial solution for visualization.

At the same time, DataV also subtly spreads its technical ideas to external developers through technical documents, and relies on the massive use of data under the platform to provide developers with true and accurate improvement directions. Not only does the end user feel the value of data visualization, but he also hopes to pass this value to thousands of developers who are struggling for visualization.

Another major direction of DataV is that it will continue to strengthen investment in the research and exploration direction of visual analysis, optimize the product framework to support more powerful data interaction functions, and strengthen cooperation with scientific research institutions such as universities on the basis of visualization theory. Ensure that you have more powerful and lasting power support in the direction of data visualization.