[Python] Several ways to resolve a domain name from a URL

Source: Internet
Author: User
Tags tld

Several ways python resolves a domain name from a URL

Find the domain name from the URL, the first thought is to use the regular, and then look for the corresponding class library. With regular parsing there are many incomplete places, the URL has a domain name, the domain name suffix has been increasing and so on. Through Google to find several methods, one is to use the Python module and the regular combination to resolve the domain name, the other is to enable the third party to write the parsing module directly parse out the domain name.

The URL to resolve
urls = [" http://meiwen.me/src/index.html "," http://1000chi.com/game/index.html ", "Http://see.xidian.edu.cn/cpp/html/1429.html", "https://docs.python.org/2/howto/regex.html", "" "https:/ /www.google.com.hk/search?client=aff-cs-360chromium&hs=tsj&q=url%e8%a7%a3%e6%9e%90%e5%9f%9f%e5%90%8dre &OQ=URL%E8%A7%A3%E6%9E%90%E5%9F%9F%E5%90%8DRE&GS_L=SERP.3 ... 74418.86867.0.87673.28.25.2.0.0.0.541.2454.2-6j0j1j1.8.0 .... 0...1c.1j4.53.serp. 26.2.547.IUHTJ4UOYHG "" "," file:///D:/code/echarts-2.0.3/doc/example/tooltip.html "," HTTP://API.MONGODB.O Rg/python/current/faq.html#is-pymongo-thread-safe "," https://pypi.python.org/pypi/publicsuffix/"," http: 127.0.0.1:8000 "] 
How to use urlparse+ regular
Import refrom urlparse Import Urlparsetophostpostfix = ('. com ', '. La ', '. Io ', '. Co ', '. Info ', '. Net ', '. org ', '. Me ', '. mobi '    , '. us ', '. Biz ', '. xxx ', '. Ca ', '. co.jp ', '. com.cn ', '. net.cn ', '. org.cn ', '. mx ', '. tv ', '. ws ', '. Ag ', '. Com.ag ', '. Net.ag ', '. Org.ag ', '. am ', '. Asia ', '. at ', '. is ', '. com.br ', '. net.br ', '. Bz ', '. com.bz ', '. net.bz ', '. CC ', '. com.co ', '. net.co ', '.    Nom.co ', '. de ', '. Es ', '. com.es ', '. nom.es ', '. org.es ', '. eu ', '. FM ', '. Fr ', '. GS ', '. In ', '. co.in ', '. firm.in ', '. Gen.in ', '. Ind.in ', '. net.in ', '. org.in ', '. it ', '. Jobs ', '. JP ', '. Ms ', '. com.mx ', '. nl ', '. Nu ', '. co.nz ', '. net.nz ', '. org.nz ', '. Se ', '. TC ', '. tk ', '. tw ', '. com.tw ', '. idv.tw ', '. org.tw ', '. HK ', '. co.uk ', '. ' me.uk ', '. org.uk ', '. VG ', '. com.hk ', regx = R ' [ ^\.] + (' + ' | '). Join ([H.replace ('. ', R ' \. ') for h in Tophostpostfix]) + ') $ ' pattern = Re.compile (regx,re. IGNORECASE) print "--" *40for url in urls:parts = Urlparse (URL) host = Parts.netloc m = Pattern.search (host) re s = M.group () if M else Host print "UNKONW" if not res else res

The results of the operation are as follows:

meiwen.me1000chi.comsee.xidian.edu.cnpython.orggoogle.com.hkunkonwmongodb.orgpython.org127.0.0.1:8000

It's basic to accept

Urllib to parse the domain name
import urllibprint "--"*40for url in urls:    proto, rest = urllib.splittype(url)    res, rest = urllib.splithost(rest)    print "unkonw" if not res else res

The results of the operation are as follows:

meiwen.me1000chi.comsee.xidian.edu.cndocs.python.orgwww.google.com.hkunkonwapi.mongodb.orgpypi.python.org127.0.0.1:8000

will bring the www. Also need further analysis to

Using a third-party module TLD
from tld import get_tldprint "--"*40for url in urls:    try:        print  get_tld(url)    except Exception as e:        print "unkonw"

Operation Result:

meiwen.me1000chi.comxidian.edu.cnpython.orggoogle.com.hkunkonwmongodb.orgpython.orgunkonw

The results are acceptable.

Other parsing modules that you can use:
    • Tld
    • Tldextract
    • Publicsuffix


This article is derived from "Orangleliu notebook " Blog, be sure to keep this source http://blog.csdn.net/orangleliu/article/details/39545821

[Python] Several ways to resolve a domain name from a URL

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.