In the past one or two years, the short video business has been flourishing, and each video website has its own special short video content. If a program can download the latest videos from popular users on major video websites, it not only facilitates viewing, but also publishes copyrighted videos on personal social websites, how nice is it to increase your popularity? in the past one or two years, the short video business has been booming. each video website has its own special short video content. If a program can download the latest videos from popular users on major video websites, it not only facilitates viewing, but also publishes copyrighted videos on personal social websites, how nice is it to increase your popularity?
In the past one or two years, the short video business has been flourishing, and each video website has its own special short video content. If you have such a program, you can download the latest videos from popular users on major video websites, which is not only convenient for you to watch, you can also publish copyrighted videos on personal social networking websites to increase your popularity?
Parker is such a project (Project address: https://github.com/LiuRoy/parker) that uses the celery framework to regularly crawl the user's video list and asynchronously download the latest video through you-get, you can easily implement distributed deployment. Because the page layout and interfaces of each website are updated frequently, Statsd monitoring is added to ensure high availability of the program to facilitate timely error detection.
Code Architecture
Currently, parker only implements download for Site B and second beats. from the framework diagram, we can see that two asynchronous interfaces are required for each type of website: parse the playback address of the published video from the user's video homepage and download the video based on the playback address. Therefore, you do not need to modify the original code to add a new resolution and download interface. I have not implemented the subsequent operations after the video is downloaded. you can freely implement the operations as needed.
During running, celery regularly sends the configured high-quality user list to the resolution interface of the corresponding website for asynchronous execution, filters out the latest video playback address, and submits it to the corresponding download interface for asynchronous download, after the download is complete, call the subsequent operations asynchronously. Therefore, you need to start a celery beat process to send scheduled tasks and several celery asynchronous tasks for parsing and downloading operations. for large videos, downloading may take a considerable amount of time, we recommend that you allocate the number of asynchronous tasks based on the number of tasks in the task list.
Program running
It has been verified that this program can run normally in ubuntu and mac. because celery in local windows cannot be started normally, it has not been verified in windows.
Dependency Library installation
The python version is 3.5. after entering the Project Directory, run:
Pip install-r requirements.txt
Create a database table
Create two tables (SQL: https://github.com/LiuRoy/parker/blob/master/spider/models/tables. SQL) in the database in advance)
Parameter configuration
Logging. yaml, params. yaml, and sites. yaml under the config path correspond to log configuration, running parameter configuration, and popular user configuration respectively.
Log configuration
In debug mode, logs are directly output to the standard output stream. in release mode, logs are output to files. Therefore, you need to configure the output log file.
Run configuration
-
Mode debug mode. In this mode, logs are directed to the standard output without monitoring data. in release mode, logs are output to the specified file with monitoring data.
-
Broker_url corresponds to the BROKER_URL of celery. it can be set to redis or rabbitmq.
-
Mysql_url: The database address. two tables must be created in advance.
-
Download_path: video download path
-
Statsd_address monitoring address
-
Video_number_per_page resolves the number of video playback addresses from the user's video homepage each time, because most users only need to set a small value for each published video. During the initial operation, you will not download a large number of videos that have been used for a long time.
-
Download_timeout video download timeout
Popular user configuration
Parker generates a list of celery beat schedat based on this configuration.
-
The name rule is <网站类型> - <任务id> , Parker uses this as the name of the scheduler task
-
Url user's publish video homepage
-
Celery parsing asynchronous task corresponding to task
-
How many minutes does minute check the user's video list?
Start a task
Go to the project directory and run the following command to start celery worker.
celery -A spider worker
Run the following command to start the celery beat scheduled task:
celery -A spider beat
Monitoring
We strongly recommend that you use a docker image provided by Amway with a monitoring environment in one minute. Then, you only need to add the hitting data for successful execution and execution exceptions to conveniently monitor whether the program runs normally.
The above is the details of the sample code for implementing the video download function using Python. For more information, see other related articles in the first PHP community!