Problems and solutions that Java may encounter when crawling Web content, generating files in Mobi format supported by the Kindle

Issue 1: Content fetching batch when parsing RSS, some sites have published time in the RSS, but the published time is always null.

Cause: The RSS release time format for those sites does not conform to ROME.JAR requirements and cannot be parsed, so returns NULL.

Solution: Modify the source code of Rome.jar, add in Com/sun/syndication/rome.properties
The Datetime.extra.masks property, which is used to format multiple release dates.
such as: Datetime.extra.masks=yyyy-mm-dd hh:mm:ss|yyyy-mm-dd hh:mm| EEE, DD MMM yy hh:mm:ss|yyyy-mm-dd HH:mm:ss. Sss
Modify the source code and then re-hit the jar package, reference to the Java project, you can parse the release time of different formats.

-------------------------------------------------------------------------------------------

Issue 2: When crawling certain Web pages, you will be prompted to encode errors, such as:
org.htmlparser.util.EncodingChangeException: Character mismatch (NEW: [0xFEFF]! = old: [0x9518 Nobelium]) for Encoding change from iso-8859-1 to GBK at character offset 214

Cause: If the title of Chinese on the page returned by the request URL is written in front of the meta, and the CharSet encoded information (GBK) set in this meta is not the same as the parser default encoding (iso-8859-1), Then you will report the mistake.
But if you don't have any Chinese before meta, you won't be able to report this mistake. Because the htmlparser itself will be encoded according to the CharSet information in the HTML source code.

Solution: After catching this exception, use the CharSet encoding set in meta to fetch it again.

-------------------------------------------------------------------------------------------

Issue 3:epub generate batch when you use the Kindlegen tool to convert an epub file to a Mobi file in a Linux environment, no exception occurs, but no Mobi file is generated.

Reason: Unknown

Solution:

Process process = Runtime.getruntime (). EXEC (Kindlegen command); InputStream is = Process.getinputstream (); int R;while ( -1! = (r = Is.read ())) {    System.out.print ((char) r);}

After you print out the value of the InputStream, you can generate the Mobi file.

-------------------------------------------------------------------------------------------

Question 4: Crawling Web content sometimes contains the crawled Web site's own JS function, video links and other code, which can be displayed on the browser may prompt JS error

solution: Filter out all tags such as script, style, link, and all tag events (such as onclick, onmouseover, etc.)

-------------------------------------------------------------------------------------------

Issue 5: The generated Mobi file cannot be recognized as an ebook on the Kindle Paperwhite device

Cause: 1. The content source directory file is missing and the catalog file is defined in the Guide node of the Opf file, so the Kindle device cannot display its directory.
2, the Kindle Paperwhite will be the newspaper, magazine type Mobi file automatically converted to Pobi file, other types of Mobi files will not be converted.

Solution: 1. Add a reference to the directory in the content source directory file and the Opf file
2, to let the Kindle Paperwhite to identify the newspaper, magazine type of Mobi file, must have the following settings:
1. Add the following code to the metadata of the Opf file:

<x-metadata> <output encoding= "Utf-8" content-type= "Application/x-mobipocket-subscription-magazine" > </output></x-metadata>

2, in Ncx in accordance with the requirements of Navpoint written

<navpoint class= "Periodical" >          <navpoint class= "section" >              <navpoint class= "article" > ......

These classes must be written, and the value of class should be periodical ,Section and Article otherwise not .

Problems and solutions that Java may encounter when crawling Web content, generating files in Mobi format supported by the Kindle