Heritrix 3.1.0 source code parsing (18)

From the beginning of this article, we will analyze the source code related to the heritrix3.1.0 system processor. In the heritrix system, the processed crawler Luri Curi object is processed by the processor in the system before being final.

Because there are a lot of processors, apart from the logic of the inheritance layer of the processor itself, the processor with similar functions is put into the same processor chain in the system.

Heritrix3.1.0 is logically abstracted as two processor chains (fetchchain and dispositionchain, and candidatechain logically belongs to dispositionchain)

Let's take a look at the UML diagram of the processor chain and processor.

The above is a static class diagram. The processor chain processorchain maintains a certain number of processor aggregation, and the processor chain processorchain implements the iterator <E> interface.

In actual system running, the system logic is abstracted as two processor chains (fetchchain and dispositionchain, and candidatechain logically belongs to dispositionchain)

Let me explain it.

Processor chain fetchchain (Org. archive. modules. fetchchain) processor (the URL seed is slightly different, which will be analyzed later ):

Org. archive. crawler. prefetch. preselector
Org. archive. crawler. prefetch. preconditionenforcer
Org. archive. modules. Fetcher. fetchdns
Org. archive. modules. Fetcher. fetchhttp
Org. archive. modules. Extractor. extractorhttp
Org. archive. modules. Extractor. extractorhtml
Org. archive. modules. Extractor. extractorcss
Org. archive. modules. Extractor. extractorjs
Org. archive. modules. Extractor. extractorswf

Processor chain (Org. archive. modules. dispositionchain) processor:

Org. archive. modules. Writer. mywriterprocessor
Org. archive. crawler. postprocessor. candidatesprocessor
Org. archive. crawler. postprocessor. dispositionprocessor

Processor corresponding to candidatesprocessor (Org. archive. modules. candidatechain) in actual operation

Org. archive. crawler. prefetch. candidatescoper
Org. archive. crawler. prefetch. frontierpreparer

If we change to the object dynamic graph that is actually running in the system, we can see that this is an imperfect composite mode combined with the iterator mode. Why is it imperfect?

Because processorchain and processor of the processor do not implement the same interface (in fact, they are all process methods [method signatures are different], and the branches and leaf nodes contain the same operation method)

Let's familiarize ourselves with the processorchain method of the processor chain.

This class implements the iterable <processor> interface, which overwrites the aggregation of the processors maintained by the iterator () method iteration. The related methods are as follows:

KeyedProperties kp = new KeyedProperties();    public KeyedProperties getKeyedProperties() {        return kp;    }        public int size() {        return getProcessors().size();    }    public Iterator<Processor> iterator() {        return getProcessors().iterator();    }    @SuppressWarnings("unchecked")    public List<Processor> getProcessors() {        return (List<Processor>) kp.get("processors");    }    public void setProcessors(List<Processor> processors) {        kp.put("processors",processors);    }

The most important method is the void process (crawluri Curi, chainstatusreceiver thread) iteration processor, and calls its process method to process the crawler Curi object.

public void process(CrawlURI curi, ChainStatusReceiver thread) throws InterruptedException {        assert KeyedProperties.overridesActiveFrom(curi);        String skipToProc = null;                        ploop: for(Processor curProc : this ) {            if(skipToProc!=null && !curProc.getBeanName().equals(skipToProc)) {                continue;            } else {                skipToProc = null;             }            if(thread!=null) {                thread.atProcessor(curProc);            }            ArchiveUtils.continueCheck();                                                ProcessResult pr = curProc.process(curi);            switch (pr.getProcessStatus()) {                case PROCEED:                    continue;                case FINISH:                    break ploop;                case JUMP:                    skipToProc = pr.getJumpTarget();                    continue;            }        }            }

The implementation class of the chainstatusreceiver thread interface is toethread (callback void atprocessor (processor proc)

Processresult PR is the processing result, which encapsulates the enumeration type and has three values.

public enum ProcessStatus {        /**         * The URI was processed normally, and no special action needs to          * be taken by the framework.         */        PROCEED,        /**         * The Processor believes that the ProcessorURI is invalid, or          * otherwise incapable of further processing at this time. The          * chain should skip subsequent processors, returning the URI.         */        FINISH,        /**         * The Processor has specified the next processor for the URI.  The          * china should skip forward to that processor instead of the reguarly         * scheduled next processor.         */        JUMP,    }

Processorchain has three inheritance classes: fetchchain, dispositionchain, and candidatechain.

The three did not cover any method. heritrix3.1.0 is probably for the processor chain processorchain's logical grouping of processors.

---------------------------------------------------------------------------

This series of heritrix 3.1.0 source code parsing is self-original

Reprinted please indicate the source of the blog garden hedgehog gentle

This article link http://www.cnblogs.com/chenying99/archive/2013/04/23/3036879.html