python3.6+beautifulsoup4.2 Crawl app apps and download app packs

---------------Environment Configuration---------------

1. Install Python-3.6.4-amd64.exe under the Windows operating system

2. Configure Environment variables

Path variable: such as C:\Users\Administrator\AppData\Local\Programs\Python\Python36\

Open the CMD Command window, enter: Python command, you can go to the Python edit command line.

3, the beautifulsoup4-4.6.0.tar.gz decompression, placed in the python36\ directory, in cmd ... \python36\beautifulsoup4-4.6.0\ execute command

Python setup.pyinstall, installation success without error.

4. Put get-pip.py in ... \python36\ directory, execute command: Python get-pip.py, install pip successfully.

5, in the \python36\scripts directory, open cmd, import requests, Pymysql, Urllib and other modules, execute the command as follows

Pip installrequests

Pip Installpymysql

Pip Installurllib

Test the Python environment---import several packages without error

Import requests

From BS4 Importbeautifulsoup

Import Pymysql

Fromurllib.request Import Urlopen

-------------Perform app crawl-----

1, first the app crawl script getmiappinfo the entire folder in ... \python36\lib\site-packages directory

2. Modify the database configuration file

\getallappinfo\baseconfig.ini

As follows:

[db]

db_host=192.168.2.73

db_port=3306

Db_username=root

Db_password=cecgw

Db_basedata=app

3, need to create a MySQL database, database name for the app

Execute table creation Script

------------------------------

--Tablestructure for Appinfo1

------------------------------

DROP TABLE ifexists ' appinfo1 ';

CREATE TABLE ' Appinfo1 ' (

' id ' int (one) not NULL auto_increment,

' channel ' varchar DEFAULT NULL,

' Faclassname ' varchar (255) DEFAULT NULL,

' subclassname ' varchar DEFAULT NULL,

' AppName ' varchar (255) DEFAULT NULL,

' Score ' varchar DEFAULT NULL,

' loadcounts ' varchar DEFAULT NULL,

' Size ' varchar (DEFAULT NULL),

' Author ' varchar (255) DEFAULT NULL,

' updatetime ' varchar DEFAULT NULL,

' Version ' varchar DEFAULT NULL,

' supsystem ' varchar DEFAULT NULL,

' Suplang ' varchar DEFAULT NULL,

' AppUrl ' varchar (255) DEFAULT NULL,

' Loadurl ' varchar (255) DEFAULT NULL,

' str1 ' varchar (255) DEFAULT NULL,

' str2 ' varchar (255) DEFAULT NULL,

' STR3 ' varchar (255) DEFAULT NULL,

' STR4 ' varchar (255) DEFAULT NULL,

' STR5 ' varchar (255) DEFAULT NULL,

PRIMARY KEY (' id ')

) engine=innodbauto_increment=1038 DEFAULT Charset=utf8;

4. Run the app crawl script

Open the. \python36\lib\site-packages\getallappinfo\ directory, open cmd,

Execute command python runmain.py

Input channel designator according to channel description

If you enter 1, automatically perform the ' Mobile assistant ' channel app information crawl

After storing to the database, download the app.apk package to \getallappinfo\pkg according to Loadurl needs, execute Python appdownload.py

Attention:

Source code See

Link: https://pan.baidu.com/s/1ggOJl2j Password: 0auz

python3.6+beautifulsoup4.2 Crawl app apps and download app packs