Sample code for video download using Python

Source: Internet
Author: User
In the past one or two years, the short video business has been flourishing, and each video website has its own special short video content. If a program can download the latest videos from popular users on major video websites, it not only facilitates viewing, but also publishes copyrighted videos on personal social websites, how nice is it to increase your popularity? in the past one or two years, the short video business has been booming. each video website has its own special short video content. If a program can download the latest videos from popular users on major video websites, it not only facilitates viewing, but also publishes copyrighted videos on personal social websites, how nice is it to increase your popularity?

In the past one or two years, the short video business has been flourishing, and each video website has its own special short video content. If you have such a program, you can download the latest videos from popular users on major video websites, which is not only convenient for you to watch, you can also publish copyrighted videos on personal social networking websites to increase your popularity?

Parker is such a project (Project address: https://github.com/LiuRoy/parker) that uses the celery framework to regularly crawl the user's video list and asynchronously download the latest video through you-get, you can easily implement distributed deployment. Because the page layout and interfaces of each website are updated frequently, Statsd monitoring is added to ensure high availability of the program to facilitate timely error detection.

Code Architecture

Currently, parker only implements download for Site B and second beats. from the framework diagram, we can see that two asynchronous interfaces are required for each type of website: parse the playback address of the published video from the user's video homepage and download the video based on the playback address. Therefore, you do not need to modify the original code to add a new resolution and download interface. I have not implemented the subsequent operations after the video is downloaded. you can freely implement the operations as needed.

During running, celery regularly sends the configured high-quality user list to the resolution interface of the corresponding website for asynchronous execution, filters out the latest video playback address, and submits it to the corresponding download interface for asynchronous download, after the download is complete, call the subsequent operations asynchronously. Therefore, you need to start a celery beat process to send scheduled tasks and several celery asynchronous tasks for parsing and downloading operations. for large videos, downloading may take a considerable amount of time, we recommend that you allocate the number of asynchronous tasks based on the number of tasks in the task list.

Program running

It has been verified that this program can run normally in ubuntu and mac. because celery in local windows cannot be started normally, it has not been verified in windows.

Dependency Library installation

The python version is 3.5. after entering the Project Directory, run:

Pip install-r requirements.txt

Create a database table

Create two tables (SQL: https://github.com/LiuRoy/parker/blob/master/spider/models/tables. SQL) in the database in advance)

Parameter configuration

Logging. yaml, params. yaml, and sites. yaml under the config path correspond to log configuration, running parameter configuration, and popular user configuration respectively.

Log configuration

In debug mode, logs are directly output to the standard output stream. in release mode, logs are output to files. Therefore, you need to configure the output log file.

Run configuration

  • Mode debug mode. In this mode, logs are directed to the standard output without monitoring data. in release mode, logs are output to the specified file with monitoring data.

  • Broker_url corresponds to the BROKER_URL of celery. it can be set to redis or rabbitmq.

  • Mysql_url: The database address. two tables must be created in advance.

  • Download_path: video download path

  • Statsd_address monitoring address

  • Video_number_per_page resolves the number of video playback addresses from the user's video homepage each time, because most users only need to set a small value for each published video. During the initial operation, you will not download a large number of videos that have been used for a long time.

  • Download_timeout video download timeout

Popular user configuration

Parker generates a list of celery beat schedat based on this configuration.

  • The name rule is <网站类型> - <任务id> , Parker uses this as the name of the scheduler task

  • Url user's publish video homepage

  • Celery parsing asynchronous task corresponding to task

  • How many minutes does minute check the user's video list?

Start a task

Go to the project directory and run the following command to start celery worker.


celery -A spider worker


Run the following command to start the celery beat scheduled task:


celery -A spider beat


Monitoring

We strongly recommend that you use a docker image provided by Amway with a monitoring environment in one minute. Then, you only need to add the hitting data for successful execution and execution exceptions to conveniently monitor whether the program runs normally.

The above is the details of the sample code for implementing the video download function using Python. For more information, see other related articles in the first PHP community!

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.