Key Points of video monitoring/Storage System Design

The video monitoring system consists of the following components:

No.	Module name	Functions and remarks
1	Device proxy	The system communicates with the front-end device.
2	Alarm	Receive/Store alarm information (external alarm)
3	Streaming Media	Forward video streams. Due to limited concurrent connections, multiple NICs are usually configured.
4	Device Access	The front-end device is connected to the system, generally through onvif, RTSP or SDK, the power system has its own protocol
5	Storage	It will be highlighted below
6	Linkage	Including policy definition domain execution
7	Web	Cross-platform use qtwebkit or MJPEG
8	Electronic Map	Generally, vector graphs are supported, videos are superimposed on maps, and multiple images are supported.
9	Cascade	Multi-level system connection
10	Control Service	Cloud Platform Control and Permission Management
11	Preview	Client
12	School hours	Set device time in a unified manner

 

Storage design objective: to save and find. The following two items are suitable for storage: videos are not lost and can be quickly found.

 

Storage Development Process:

Name or composition	Location	Structure Description	Compared with the previous generation's progress
PC DVR	Front end	Use a card on a PC to receive video streams and store them to a local hard disk.
Embedded DVR	Front end	Use a dedicated device to receive video streams and store them to a local hard disk.	Avoid risks caused by viruses and improper operations
Storage Server + ipsan	Backend	Centralized storage, in 1 + N mode, the server receives the stream and then sends it to the ipsan storage.	Improved reliability
Management machine + storage host	Backend	Centralized and distributed storage, in 1 + N mode, the storage host collects streams and stores them on ipsan. The management host is responsible for allocation and unified query.	Avoid bandwidth waste and reduce storage server pressure

 

Storage process: the device is connected to the streaming media, the streaming media, and the data is written to the disk.

 

Storage Design Features: write more and read less, easy to produce disk fragments. Based on experience, the disk has been cyclically covered for three rounds and the read/write efficiency has been reduced by 80%.

Methods To prevent disk fragments:

1. pre-allocated space: pre-allocated storage space in the disk, and full of "junk data" in the space, making it a continuous space. (It can be implemented through the "new disk initialization" process; or run a low-priority thread in the background to pre-allocate the thread when it is idle. After the allocation is complete, the thread exits .)

2. Reuse objects cyclically: old objects are marked as "invalid" instead of being deleted ". When the disk is full, search for "invalid Files" and write again.

 

Video storage format:

You can use the original format (not converted by the front end) or the unified format (conversion ). It is generally recommended to use a uniform format to facilitate the development of other related modules and improve the low coupling and general type. Note that conversion is not transcoding. Generally, the h264 stream encapsulated by PS is used. (PS encapsulation contains time information)

 

Note:

A pc client that uses an Intel i7 processor and NVIDIA independent graphics card may experience severe freezing when playing 16 1080p video screens. The general solution is to increase the resolution and bit rate when the number of lines is small. When the number of lines is large, the resolution and bit rate are appropriately reduced.

 

Pull Mode and push mode:

We recommend that you use the PULL mode, that is, the operation to obtain video streams is initiated by the backend (initiated by the client or storage host ). This facilitates the modularization of the Platform System and reduces the requirements for front-end devices.

 

Storage Mode:

The management machine has a globally unified database to store video indexes, including I frame timestamps and event time scales.

Each storage host has a small database that records the locally stored video files and the time when each file corresponds to the front-end device, which files have been marked as "invalid" and which are currently in use.

When the storage host is running, it must report the running status to the Administrator.

 

Allocate storage resources:

You can specify on the interface which IPC is allocated by the Administrator to the storage host.

When a storage host is started, it reports resources to the manager. The manager can automatically allocate which cameras are stored on which storage hosts based on the resources.

 

Video Retrieval:

When querying, the Administrator locates the queried data on a storage host. That is, users do not need to care about the storage host where the video files are stored.

Video query method:

1. Search by camera-time period: the most basic retrieval method.

2. Search by camera-event tags: (quasi-Intelligent Search) events generated by smartipc must be detected and stored. Event tags can be used for retrieval. (You can also manually add custom tags)

3. Search by camera-content: Call the analysis module when you need to search, set the analysis parameters on the Analysis Server, filter the video in advance, and then deliver the clip to the user. (This provides a new idea for video analysis !)

 

Disaster Tolerance and exception handling:

The goal is to ensure video integrity in case of exceptions. It includes three points: switchover, redundancy, and recovery.

1. Storage hosts can be divided into duty machines and backup machines, which can be set to n + 1. When a duty machine goes down, it is discovered by the management machine and used on top of the backup machine.

2. Multiple hard disks in the same storage host can be set to n + 1 redundancy. When a hard disk is damaged, use the backup disk as the top.

3. Restore the stored data after the device goes down. The simplest way is to use multi-copy storage.

4. Consider that some key videos cannot be overwritten. You can lock the database or migrate the key videos to an independent storage host.

5. troubleshooting after fault recovery: the original standby machine that has been on duty is on duty. The original host recovered from the fault is used as the backup host.

6. When an IPC, disk, or storage host fault is detected, remind the staff to handle the fault.

 

The status of the nvr, DVR, and external storage medium (such as SD card and external hard disk) in the existing storage system should be considered for the upgrade of the old system.

 

Others:

Storage Policy: You can reduce the frame rate during storage based on your needs (or suggest to you. Some cameras with relatively static images and dim light can adopt a lower frame rate.

Generally, the Storage Frame Rate and preview frame rate are consistent without considering the Sub-code stream of the main code stream.

The size of the stored data file should not be too large. The experience should be 128 MB.

Network Utilization Rate: in experience, the network utilization rate exceeds 50%, and obvious problems may occur.

Smooth switching of video files: Pre-read the video files to the streaming media for processing (try an experiment by yourself ).

In a large-scale distributed storage system, you can deploy the storage host and device in the same LAN, and deploy the management machine on the master site to solve the storage bandwidth utilization problem.

Key Points of video monitoring/Storage System Design