Making Sina Weibo crawler with Python

Morning brush space to find a lot of people recently birthday AH ~

Think about it, it's like April is a special lot of people's birthday "like me

So what are the distribution rules of each person's birthday month? Suddenly want to write a small program statistics

The most easy-to-get birthday database is probably Sina Weibo:

But the computer version of Sina Weibo is obviously a dynamic Web page ... If you want to climb this should be to parse the JS script "like the last crawl NetEase cloud music. However, there is no solution

In fact, there are more efficient ways: Crawl mobile version

Mobile version because most of the limitations of mobile browser are simplified, more conducive to crawler

Note The above URL: http://weibo.cn/5156584529/info

Tested by different users is only the middle of the number is different, so long as the enumeration of numbers can be implemented crawler ~

But mobile Weibo wants to see the user profile is required to log in. So we have to simulate the login, get the cookie, then visit the URL, access to user information.

Many websites use cookies for login, and the general process is as follows:

The user enters the user name password, the browser submits these form form (form) to the server, if the server determines that the user name password is correct then returns a cookie, then the browser will record this cookie. Then use the local cookie again to access it without having to log in.

Demo Login:

Open the Weibo mobile home page http://weibo.cn, click Login to get the login address:

http://login.weibo.cn/login/?ns=1&revalid=2&backURL=http%3A%2F%2Fweibo.cn%2F&backTitle=%CE%A2%B2% a9&vt=

"This interface is really ugly ...

Enter your username and password to sign in, grab a package with chrome and view the form:

In fact, we just need the form to be enough.

Using URLLIB2 in Python, use the form data to access the login page, get the cookie, and then use the cookie to access the user page.

But one more thing to note: Sina Weibo has been anti-crawler, so it will encounter this error:

Urllib2. Httperror:http Error 403:forbidden

So we're going to add a header message headers to impersonate a browser.

Code

1 __author__='IBM'2 ImportUrllib23 ImportUrllib4 ImportCookielib5headers = {'user-agent':'mozilla/5.0 (Windows; U Windows NT 6.1; En-us; rv:1.9.1.6) gecko/20091201 firefox/3.5.6'}6Cookie =Cookielib. Cookiejar ()7Opener =Urllib2.build_opener (urllib2. Httpcookieprocessor (cookie))8 9 #uurl= ' Http://weibo.cn/5156582529/info 'TenWurl='Http://login.weibo.cn/login/?backURL=&backTitle=&vt=4&revalid=2&ns=1' One  ALogindata=Urllib.urlencode ( -     { -         'Mobile':'don't peek at my cell phone number! ', the         'password_8199':'don't peek at my code! ', -         'Remember':' on', -         'Backurl':'http%253a%252f%252fweibo.cn%252f', -         'Backtitle':'%e5%be%ae%e5%8d%9a', +         'Trycount':"', -         'VK':'8199_4012_2261332562', +         'Submit':'%e7%99%bb%e5%bd%95' A     } at ) -  -loginreq=Urllib2. Request ( -Url=Wurl, -Data=Logindata, -headers=Headers in ) -  tologinres=Opener.open (loginreq) + PrintLoginres.read () -  theHtml=opener.open (URLLIB2. Request (url='Http://weibo.cn/5156584529/info', headers=headers)) *dat=Html.read () $ PrintDat

View Code

The output dat is the HTML of the user's profile page. Whatever information you want, you can go inside and find it.

"But there's still one problem: Watch out for the two red underline in the form:

The two numbers tested are different every time you log in. And the same number is valid, that is, after a while this code may not be able to login ...

Personally guess this may be for the anti-reptile bar ...

Under construction

REF:

http://blog.csdn.net/pleasecallmewhy/article/details/9305229

http://www.douban.com/note/131370224/

Making Sina Weibo crawler with Python