WebMagic's design references the best crawler scrapy in the industry.

Http://webmagic.io/docs/zh/posts/ch1-overview/thinking.html

Https://github.com/psvehla/liferay-spring-mvc-portlet

http://www.huqiwen.com/2012/08/30/liferay-6-1-development-study-1/

.1 Design ideas of WebMagic

1. A framework, an area

A good framework must be a cohesive field of knowledge. WebMagic's design refers to the industry's best crawler scrapy, while the implementation of the HttpClient, Jsoup and other Java world's most mature tools, the goal is to do a Java language web crawler-like implementation.

If you are a reptile developer, then WebMagic will be very easy to use, it almost uses the Java native development method, but provides some modular constraints, encapsulates some cumbersome operations, and provides some convenient features.

If you are a novice crawler developer, then using and understanding WebMagic will let you understand the common patterns of crawler development, Toolchain, and how to handle problems. After skillful use, it is not difficult to believe that you are developing a crawler from scratch.

Because of this goal, the core of WebMagic is very simple-in this case, the functionality is to give in to the simplicity of the concession.

2. Micro-core and high scalability

WebMagic consists of four components (Downloader, Pageprocessor, Scheduler, Pipeline), the core code is very simple, mainly to combine these components and complete the multi-threaded task. This means that in webmagic, you can basically customize the functionality of the crawler.

The core of WebMagic in the Webmagic-core package, the other packages you can understand as an extension to webmagic-this is no different from writing an extension as a user.

3. Focus on practicality

While the core needs to be simple enough, WebMagic also implements a number of convenient features that can help develop in an extended way. For example, the development of crawler based on annotation pattern, as well as the extension of XPath syntax xsoup and so on. These features are optional in webmagic, and their development goal is to make the user's crawler as simple as possible and easy to maintain.