The asyncio asynchronous coroutine framework is used to collect live broadcast bullet screens of Site B and asyncio

The asyncio asynchronous coroutine framework is used to collect live broadcast bullet screens of Site B and asyncio

Preface

Although the title is full site, only full-day bullet screens of top 100 live broadcasting rooms are collected.

The bullet screen collection system was modified based on the Python version of the previous B-site live video bullet screen. For specific protocol analysis, see the previous article.

The Live bullet screen protocol is directly based on the TCP protocol. Therefore, it is difficult for site B to counter such behaviors. There should be technical means I don't know to detect such malicious behaviors like me.

I tried connecting 100 rooms at the same time, and connecting a single room for 100 experiments, there is no problem.> 150 will be closed.

Live broadcasting room Selection

Now the bullet screen collection system is relatively simple in selecting the live broadcasting room, and the top levels are selected directly.

Later will modify this part, change to timing to http://live.bilibili.com/all to view the new live broadcasting room, and dynamic add task.

Asynchronous task and bullet screen Storage

The collection system still uses the asyncio asynchronous coroutine framework, and adds the following methods to the loop for each live broadcasting room.

danmuji = bilibiliClient(url, self.lock, self.commentq, self.numq)task1 = asyncio.ensure_future(danmuji.connectServer())task2 = asyncio.ensure_future(danmuji.HeartbeatLoop())

In fact, if you put HeartbeatLoop in connectorServer to start the HeartbeatLoop task, the code looks more elegant. But this is because I need to maintain a task list, which will be described later.

I spent some time on bullet screen storage.

Database storage is a process of synchronizing I/O, And the Insert operation will block the bullet screen collection tasks. Although there is an asynchronous interface such as aiomysql, it is too troublesome to configure the database. I imagine that this small system can be conveniently deployed.

Finally, I chose to use the built-in sqlite3. However, sqlite3 cannot perform parallel operations, so a thread is enabled for separate database storage. In another thread, 100*2 Tasks collect all the bullet screens and number of people, and insert them into the queue commentq and numq. The storage thread wakes up every 10 s, writes data in the queue into sqlite3, and clears the queue.

Network traffic is not blocked with the combination of multithreading and Asynchronization.

Possible connection failure scenarios

The bullet screen protocol is directly based on TCP, And the bit and bit are directly correlated. Once a parsing error occurs, it is easy to throw an Exception (personally, although TCP is a reliable transmission, but it is also possible that the B-site server has its own errors ). Therefore, it is necessary to design an automatic reconnection mechanism.

As mentioned in the asyncio document,

Done means either that a result/exception are available, or that the future was canceled.

If the function returns normally, throws an exception, or is canceled, the current task is exited. You can use done () to determine.

Each live broadcasting room corresponds to two tasks. The parsing task is the easiest task, but does not affect the heartbeat task. Therefore, you must find and end the corresponding heartbeat task.
When creating a task, use the dictionary to record the two tasks in each room,

self.tasks[url] = [task1, task2]

Check every 10 s during running,

for url in self.tasks:  item = self.tasks[url]  task1 = item[0]  task2 = item[1]  if task1.done() == True or task2.done() == True:    if task1.done() == False:      task1.cancel()    if task2.done() == False:      task2.cancel()    danmuji = bilibiliClient(url, self.lock, self.commentq, self.numq)    task11 = asyncio.ensure_future(danmuji.connectServer())    task22 = asyncio.ensure_future(danmuji.HeartbeatLoop())    self.tasks[url] = [task11, task22]

In fact, I have only seen one failed task scenario, because the host room is blocked and cannot enter the live broadcasting room.

Conclusion

1. The number of people on Site B is counted based on the number of connections to the bullet screen server. By manipulating the number of links, you can instantly increase the number of viewers. Is there a business opportunity?
2. During the past few days, we found that even if most of the rooms were not broadcast live, there were more than 5 people, including early morning. I can only guess that people like me collect bullet screens within 24 h.
3. The top100 has an average of 40 MB of bullet screen data per day.
4. What can I do with the collected bullet screens? I haven't thought about it yet, maybe I can use it for user behavior analysis-_ ^

Finally, the source code of the GITHUB address https://github.com/lyyyuna/bilibili_danmu_colloector