CM Development Log-search engine (iii)

In this blog post, for the entire life cycle of SOLR, I hope you will have a preliminary understanding (so that you can capture the context of the entire series of articles).

I summarize it in my own words as follows: SOLR Multi-core creation , preprocessing of indexes, and index creation and persistence (the Search API interacts with index files)

The wrong place wants everyone to correct me. In addition, recently looking at some more detailed information about SOLR, found that many of the online information is either a simple copy of each other, or is the English material. Always feel less comfortable, some things you still have to see the SOLR wiki or view the source code. Also hope that their own little research, can give learning application Solr,lucene or WCS friends a little bit of reference. In addition, the Internet and e-commerce more interested in friends can also be on the e-commerce site development process has a general understanding.

Back to the text, the previous blog post described the creation of SOLR multi-core based on WCS. Create SOLR multicore utility execution The whole process not too much attention, after all, is a framework package of logic, through the phenomenon of the essence is the most serious! I think the focus of SOLR multicore is the understanding of the concept: to create the appropriate SOLR kernel based on the type and characteristics of the background data of your website system.

For example, in the WCS Web site system, backstage has a variety of miscellaneous data, but we can according to the data application scenario to divide them:

-is a product, user, order, inventory and other data itself, we call the Entity data, this large class can be divided into two small categories

-the entity itself, such as the name of the coffee machine, manufacturer, performance, price, etc., these are called structured data

-Is the entity attached, or the coffee machine For example, the coffee machine attached to the product manuals, policy information, etc., these are called unstructured data

-For classification purposes we call categorical data

In conclusion, we divide the complex data of WCS into 3 categories, namely structured Entity Data ,unstructured entity data and classified data .

The above is another summary of the SOLR multicore, which goes to the next loop: index preprocessing

Index preprocessing

First, for the original eco-SOLR, he supported the data being written into SOLR memory in the form of Xml,csv,json. This data can have 100, if your memory is large enough, it can be 1000 or more. However, more scenarios are indexed by directly reading the database through a configuration file. The process of reading a database and creating a temporary table, which is the preprocessing of the index.

Let's start by understanding what an index is, and then naturally understand what an index preprocessing is.

Index, my understanding is:

1. First he is a file (one or more)

2. These files are equivalent to the database table, but the indexed field is called Domain (field), and the corresponding domain (field) is mapped according to the database table fields.

3. After the corresponding, the index file is like a database table: He has only one primary key, and there are many fields. We are able to find the information we want only through this primary key, thus eliminating the process of checking the database table from the left outer connection.

For example, our website has brand information, and the brand information table has a foreign key associated with the main commodity table.

So in order to create an index on the brand, we first have to have a temporary table ti_brands, which defines the information we need to index, such as Product ID, name, initials, etc.

Second, we have to sort out the database fields we need to index (by the way of the left join Main table), so we have the field information we need and save it all in the temp table ti_brands.

Inside the WCS, the process of creating a TI temp table is called index preprocessing.

Additional WCS index preprocessing configuration file Wc-dataimport-preprocess-brands.xml

<?xml version= "1.0" encoding= "UTF-8"? ><_config:dihpreprocessconfig xmlns:_config= "http://www.ibm.com/ Xmlns/prod/commerce/foundation/config "xmlns:xsi=" Http://www.w3.org/2001/XMLSchema-instance "xsi:schemalocation= "Http://www.ibm.com/xmlns/prod/commerce/foundation/config. /.. /xsd/wc-dataimport-preprocess.xsd "> <!--Get catalog IDs for Catentry-<_config:data-processing -config processor= "Com.ibm.commerce.foundation.dataimport.preprocess.BootsStaticBrandsDataPreProcessor" Batchsize= "><_config:table definition=" CREATE table Ti_brands (bdc_brand_id number,bdc_brand_name VARCHAR2 (255 byte), Brand_name_first_char VARCHAR2 (255 byte), Premium_brand_flag VARCHAR2 (255 byte), Brand_treatment_flag VARCHAR2 (255 byte), Display_flag VARCHAR2 (255 byte), Brand_room_url VARCHAR2 (255 byte), Brand_treatment_url VARCHAR2 ( 255 BYTE), PRIMARY KEY (bdc_brand_id)) "Name=" Ti_brands "/><_config:query sql=" select T.brandid as Brandid, T.brandname as Brandname,tib. Brand_naMe_first_char as Brandnamefirstchar,tib. Premium_brand_flag as Premiumbrandflag,tib. Brand_treatment_flag as Brandtreatmentflag,tib. Display_flag as Displayflag,tib. Brand_room_url as Brandroomurl,tib.  Brand_treatment_url as Brandtreatmenturl from (select br.brand_id as Brandid, br.brand_name as brandname from BDCBRAND BR) T left joins Ti_brands Tib on (T.brandid = Tib.      bdc_brand_id) "/> <_config:mapping> <_config:key querycolumn=" Brandid "tablecolumn=" bdc_brand_id "/> <_config:column-mapping> <_config:column-column-mapping><_config:column-column querycolumn= "Bra Ndname "tablecolumn=" Bdc_brand_name "/><_config:column-column querycolumn=" Brandnamefirstchar "tableColumn=" Brand_name_first_char "/><_config:column-column querycolumn=" Premiumbrandflag "tableColumn=" PREMIUM_BRAND_ FLAG "/><_config:column-column querycolumn=" Brandtreatmentflag "tablecolumn=" Brand_treatment_flag "/>< _config:column-column querycolumn= "Displayflag" taBlecolumn= "Display_flag"/><_config:column-column querycolumn= "Brandroomurl" tableColumn= "BRAND_ROOM_URL"/ ><_config:column-column querycolumn= "Brandtreatmenturl" tablecolumn= "Brand_treatment_url"/> </_config :column-column-mapping> </_config:column-mapping> </_config:mapping> </_config:data-processin G-config></_config:dihpreprocessconfig>

CM Development Log-search engine (iii)