How can I capture dynamic URLs and parse configuration files in the nuttage?

It won't be captured during running, and the reason is analyzed: mainly in CONF/crawl-urlfilter.txt.
Analysis: If you filter files using the default configuration of nutch, do you not capture the inclusion? *! @ =
Solution: Modify the filter rule of crawl-urlfilter,

# The url filter file used by the crawl command.

# Better for Intranet crawling.
# Be sure to change my. domain. name to your domain name.

# Each non-comment, non-blank line contains a regular expression
# Prefixed by '+' or '-'. The first matching pattern in the file
# Determines whether a URL is wrongly ded or ignored. If no pattern
# Matches, the URL is ignored.

# Skip file:, ftp:, & mailto: URLs
-^ (File | FTP | mailto ):

# Skip image and other suffixes we can't yet parse
-/. (GIF | JPG | PNG | ICO | CSS | sit | EPS | WMF | zip | PPT | MPG | XLS | GZ | RPM | tgz | mov | mov | exe | JPEG | BMP) $

# Skip URLs containing certain characters as probable queries, etc.
-[? *! @ =] // Indicates filtering URLs that contain the specified characters. For example,-[~]

# Skip URLs with slash-delimited segment that repeats 3 + times, to break Loops
-. * (/. + ?) /.*? /1 /.*? /1/

# Accept hosts in my. domain. Name
+ ^ Http: // [a-z0-9] */.) * tianya.cn/#/s/s#* // filter Regular Expressions ([a-z0-9] */.) * represents any number and letter, and [/S] * represents any character
# Skip everything else
-.

Web Crawler settings
Nutch itself contains a crawler indexing the target site and a Web interface for searching. Before using the query interface, you need to set up a nutch crawler to crawl the target site.
Description of some configuration files:
Nutch/CONF/nutch-default.xml

Set HTTP. Agent. name. If HTTP. Agent. Name is empty, crawlers cannot start normally. You can set any name you like, for example, Vik-robot.
Indexer. mergefactor/indexer. minmergedocs both values are changed to 500. The larger the values of these two parameters, the higher the performance, the more memory consumed. If the value is too large, memory overflow may occur. In actual usage, the maximum memory usage under the current parameter is 3xxm.
HTTP. Timeout indicates the maximum timeout wait time. If the accessed connection does not respond for a long time, it will be discarded.
DB. Max. outlinks. Per. Page this parameter indicates the maximum number of external connections supported by a single page. If it is your internal system, set a large number.

 

Nutch/CONF/crawl-urlfilter.txt

This file is used to set the filtering relationship of URLs to be indexed. Each row has one filter condition.-indicates that the URL is not included, and + indicates that the URL is included.
[? *! @ =]. This row indicates that all dynamic URLs are not crawled. Most systems now have a lot of dynamic URLs. This filtering condition may make you unable to catch any content.
Set URL filtering relationships for each system. The specific setting method varies with different application systems. Here we use the popular forum discuz as an example. In this URL Filter, only the part list and post content are captured.

# Skip file:, ftp:, & mailto: URLs
-^ (File | FTP | mailto ):

# Skip image and other suffixes we can't yet parse
-/. (GIF | JPG | PNG | ICO | CSS | sit | EPS | WMF | zip | PPT | MPG | XLS | GZ | RPM | tgz | mov | mov | exe | JPEG | BMP) $

# Skip URLs containing certain characters as probable queries, etc.
#-[? *! @ =]

# Skip URLs with slash-delimited segment that repeats 3 + times, to break Loops
-. * (/. + ?) /.*? /1 /.*? /1/

# Accept hosts in my. domain. Name
# + ^ Http: // ([a-z0-9] */.) * My. domain. Name/
# Discuz
+ ^ Http://mysite.com/discuz/index.php$
+ ^ Http://mysite.com/discuz/forumdisplay.php? FID =/d + $
+ ^ Http://mysite.com/discuz/forumdisplay.php? FID =/d + & page =/d + $
+ ^ Http://mysite.com/discuz/viewthread.php? Tid =/d + & extra = Page % 3d/d + $
+ ^ Http://mysite.com/discuz/viewthread.php? Tid =/d + & extra = Page % 3d/d + & page =/d + $
# Skip everything else
-.

Nutch/CONF/regex-urlfilter.txt

I don't know whether the configuration file works, but it is recommended to comment out ,-[? *! @ =].