python3.6+beautifulsoup4.2 Crawl app apps and download app packs

Source: Internet
Author: User



---------------Environment Configuration---------------



1. Install Python-3.6.4-amd64.exe under the Windows operating system



2. Configure Environment variables



Path variable: such as C:\Users\Administrator\AppData\Local\Programs\Python\Python36\



Open the CMD Command window, enter: Python command, you can go to the Python edit command line.






3, the beautifulsoup4-4.6.0.tar.gz decompression, placed in the python36\ directory, in cmd ... \python36\beautifulsoup4-4.6.0\ execute command



Python setup.pyinstall, installation success without error.



4. Put get-pip.py in ... \python36\ directory, execute command: Python get-pip.py, install pip successfully.



5, in the \python36\scripts directory, open cmd, import requests, Pymysql, Urllib and other modules, execute the command as follows



Pip installrequests



Pip Installpymysql



Pip Installurllib



Test the Python environment---import several packages without error



Import requests



From BS4 Importbeautifulsoup



Import Pymysql



Fromurllib.request Import Urlopen









-------------Perform app crawl-----



1, first the app crawl script getmiappinfo the entire folder in ... \python36\lib\site-packages directory



2. Modify the database configuration file



\getallappinfo\baseconfig.ini



As follows:



[db]



db_host=192.168.2.73



db_port=3306



Db_username=root



Db_password=cecgw



Db_basedata=app



3, need to create a MySQL database, database name for the app



Execute table creation Script



------------------------------



--Tablestructure for Appinfo1



------------------------------



DROP TABLE ifexists ' appinfo1 ';



CREATE TABLE ' Appinfo1 ' (



' id ' int (one) not NULL auto_increment,



' channel ' varchar DEFAULT NULL,



' Faclassname ' varchar (255) DEFAULT NULL,



' subclassname ' varchar DEFAULT NULL,



' AppName ' varchar (255) DEFAULT NULL,



' Score ' varchar DEFAULT NULL,



' loadcounts ' varchar DEFAULT NULL,



' Size ' varchar (DEFAULT NULL),



' Author ' varchar (255) DEFAULT NULL,



' updatetime ' varchar DEFAULT NULL,



' Version ' varchar DEFAULT NULL,



' supsystem ' varchar DEFAULT NULL,



' Suplang ' varchar DEFAULT NULL,



' AppUrl ' varchar (255) DEFAULT NULL,



' Loadurl ' varchar (255) DEFAULT NULL,



' str1 ' varchar (255) DEFAULT NULL,



' str2 ' varchar (255) DEFAULT NULL,



' STR3 ' varchar (255) DEFAULT NULL,



' STR4 ' varchar (255) DEFAULT NULL,



' STR5 ' varchar (255) DEFAULT NULL,



PRIMARY KEY (' id ')



) engine=innodbauto_increment=1038 DEFAULT Charset=utf8;



4. Run the app crawl script



Open the. \python36\lib\site-packages\getallappinfo\ directory, open cmd,



Execute command python runmain.py



Input channel designator according to channel description



If you enter 1, automatically perform the ' Mobile assistant ' channel app information crawl






After storing to the database, download the app.apk package to \getallappinfo\pkg according to Loadurl needs, execute Python appdownload.py






Attention:



Source code See



Link: https://pan.baidu.com/s/1ggOJl2j Password: 0auz



python3.6+beautifulsoup4.2 Crawl app apps and download app packs


Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.