1. Use the Requests library and the BEAUTIFULSOUP4 library to crawl the time, title, link, and source of the campus news list.
1 ImportRequests#HTTP Libraries extract data from HTML or XML2 fromBs4ImportBeautifulSoup#Reptile Library BeautifulSoup43url = requests.get ("http://news.gzcc.cn/html/xiaoyuanxinwen/")4Url.encoding ="Utf-8"5Soup = BeautifulSoup (Url.text,'Html.parser')6 #print (Soup.head.title.text)7
8 #find the HTML element with a specific tag: ' ID ' prefix ' # '; ' class ' prefix '. ', other direct soup.select (' P ')9 forNewsinchSoup.select ('Li'):Ten ifLen (News.select ('. News-list-title')) >0: One #Print (News.select ('. News-list-title ')) #1. Crawled out of the data to generate a large list A #Print (News.select ('. News-list-title ') [0]) #2. Remove the first element from the list - #Print (News.select ('. News-list-title ') [0].text) #3. BS4 Library Text () is responsible for removing - theTime = News.select ('. News-list-info') [0].contents[0].text#Time -title = News.select ('. News-list-title') [0].text#title -href = News.select ('a') [0]['href']#links - +Href_text = requests.get (HREF)#Link Content -Href_text.encoding ="Utf-8" +Href_soup = BeautifulSoup (Href_text.text,'Html.parser') AHref_text_body = Href_soup.select ('. Show-content') [0].text at Print(time,title,href,href_text_body)
2. Choose a topic that interests you, do similar things, and prepare to "crawl network data and perform text analysis."
1 ImportRequests2 fromBs4ImportBeautifulSoup3Url= Requests.get ('https://www.bilibili.com')4url.encoding="Utf-8"5Soup=beautifulsoup (Url.text,'Html.parser')#' Html.parser ' is the parser for HTML6 #print (soup)7 forTitlesinchSoup.select ('. Nav-name'):8 #If Len (Titles.select ('. Nav-name ')) >0:9 #title=titles.select ('. Nav-name ') [0].textTen Print(Titles.text) OneDouga = Soup.select ('#bili_douga') A forRankinchDonga.select ('. Rank-list hot-list'): - - the ifLen (Rank.select ('. Rank-list hot-list')) >0: -Ranks=rank.select ('. Rank-item Highlight') [0].text - Print(ranks)
Does not crawl into the "animation" in this "ranking" URL.
python-web crawler