Several methods for extracting domain names from URLs using Python

Source: Internet
Author: User
Tags tld
This article mainly introduces several methods for extracting domain names from URLs using Python. This article provides three methods for extracting domain names from URLs, if you need a domain name, you can refer to the following url to find the domain name. First, you can use regular expressions and then find the corresponding class library. There are a lot of imperfections in regular expression parsing, including domain names in URLs and increasing domain name suffixes. Several methods can be found through google. one is to use the Python built-in module and the regular expression to resolve the domain name, and the other is to enable the third party to directly resolve the domain name with the prepared parsing module.

Url to be parsed

The code is as follows:


Urls = ["http://meiwen.me/src/index.html ",
"Http://1000chi.com/game/index.html ",
"Http://see.xidian.edu.cn/cpp/html/1429.html ",
"Https://docs.python.org/2/howto/regex.html ",
"Https://www.google.com.hk/search? Client = aff-cs-360chromium & hs = TSj & q = url % E8 % A7 % A3 % E6 % 9E % 90% E5 % 9F % 9F % E5 % 90% 8DE & oq = url % E8 % a7 % A3 % E6 % 9E % 90% E5 % 9F % 9F % E5 % 90% 8DE & gs_l = serp.3... 74418.86867.0.87673.28.25.2.0.0.0.541.2454.2-6j0j1j1. 8. 0 .... 0... 1c. 1j4. 53. serp .. 26.2.547.IuHTj4uoyHg """,
"File: // D:/code/echarts-2.0.3/doc/example/tooltip.html ",
"Http://api.mongodb.org/python/current/faq.html#is-pymongo-thread-safe ",
"Https://pypi.python.org/pypi/publicsuffix ",
"Http: // 127.0.0.1: 8000"
]

Use urlparse + regular expression

The code is as follows:


Import re
From urlparse import urlparse

TopHostPostfix = (
'. Com ','. la ','. io ','. co ','. info ','.. net ','. org ','. me ','. mobi ',
'. Us','. Biz', '. xxx','. ca', '.co.jp', '.com.cn', '.net.cn ',
'.Org.cn', '. mx', '. TV', '. Ws','. AG', '. com. AG','. net. AG ',
'. Org. AG','. am', '. air','. at', '. be',' .com.br ',' .net.br ',
'. Bz','. com. bz', '. net. bz','. CC', '. com. co','. net. co ',
'. Nom. co','. de', '. Els','. com. Els', '. nom. Els','. org. Els ',
'. Eu ','. fm ','. fr ','. gs ','. in ','. co. in ','. firm. in ','. gen. in ',
'. Ind. in ','. net. in ','. org. in ','. it ','. jobs ','. jp ','. ms ',
'. Com. mx', '. nl', '. nu', '. co. nz','. net. nz', '. org. nz ',
'. Se','. tc ','. tk ','. tw ',' .com.tw ',' .idv.tw ',' .org.tw ',
'. Hk', '. co. uk', '. me. uk', '. org. uk', '. vg', ".com.hk ")

Regx = R' [^ \.] + ('+' | '. join ([h. replace ('. ', r '\. ') for h in topHostPostfix]) +') $'
Pattern = re. compile (regx, re. IGNORECASE)

Print "--" * 40
For url in urls:
Parts = urlparse (url)
Host = parts. netloc
M = pattern. search (host)
Res = m. group () if m else host
Print "unkonw" if not res else res

The running result is as follows:

The code is as follows:


Meiwen. me
1000chi.com
See.xidian.edu.cn
Python.org
Google.com.hk
Unkonw
Mongodb.org
Python.org
127.0.0.1: 8000

Acceptable

Urllib to resolve domain names

The code is as follows:


Import urllib

Print "--" * 40
For url in urls:
Proto, rest = urllib. splittype (url)
Res, rest = urllib. splithost (rest)
Print "unkonw" if not res else res

The running result is as follows:

The code is as follows:


Meiwen. me
1000chi.com
See.xidian.edu.cn
Docs.python.org
Www.google.com.hk
Unkonw
Api.mongodb.org
Pypi.python.org
127.0.0.1: 8000

Www. will also be taken, and further analysis is required.

Use third-party module tld

The code is as follows:


From tld import get_tld

Print "--" * 40
For url in urls:
Try:
Print get_tld (url)
Failed T Exception as e:
Print "unkonw"

Running result:

The code is as follows:


Meiwen. me
1000chi.com
Xidian.edu.cn
Python.org
Google.com.hk
Unkonw
Mongodb.org
Python.org
Unkonw

Acceptable results

Other available parsing modules:

Tld
Tldextract
Publicsuffix

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.