Global sorting Optimization

I always think that a periodic summary is a good habit. Many things I have done by myself. If I do not summarize them in time, I will forget it after a while. When it comes to use, it takes a lot of time to get familiar with it again. So I decided to take some time to summarize some of the previous Optimizations to the international search offline system (the international search here mainly refers to the AE, SC, and SC stores, and AE is Ali.

I always think that a periodic summary is a good habit. Many things I have done by myself. If I do not summarize them in time, I will forget it after a while. When it comes to use, it takes a lot of time to get familiar with it again. So I decided to take some time to summarize some of the previous Optimizations to the international search offline system (the international search here mainly refers to the AE, SC, and SC stores, and AE is Ali.

I always think that a periodic summary is a good habit. Many things I have done by myself. If I do not summarize them in time, I will forget it after a while. When it comes to use, it takes a lot of time to get familiar with it again. So I decided to take some time to summarize some of the previous Optimizations to the international search offline system (the international search here mainly refers to the AE, SC, and SC stores, and the AE is AliExpress, SC is Sourcing, and these optimizations are common to these applications. This not only serves as a memo, but is certainly an excellent solution if it can inspire readers.

Since it is related to the search offline system, let's take a look at several major links in the full international search process, as shown in 1.

. Full Process

1) dump: reads data from the database and writes data to hbase. The database is fully dumped only when the database is full. Generally, you only need to run a small amount of data once a day, data updates in the database will update hbase incrementally.

2) join, read hbase, join multiple tables, generate a single doc, a doc contains all the fields of a product.

3) global sort, that is, global sorting. global sorting is performed for the product according to the global score of the product. The generated file is not necessarily ordered internally.

4) abuild: read the files generated after global sorting and build indexes. The generated indexes are stored on HDFS.

5) dispatch: distribute the index from HDFS to the corresponding search machine.

6) switch, switch the index, program, configuration, and algorithm dictionary, and launch the new index to provide external services.

This time, we will summarize the global sorting optimization. Any project or requirement has a corresponding background. Why do we need to perform global sorting in offline computing?

Speaking of this, hierarchical search is introduced. In earlier times, when the International Site Search Engine provided external services, all segments will be queried when processing each search request, but for each request, you only need to return a certain number of result sets. Therefore, it is not necessary to query all segments, but it will only cause performance loss. As a result, layered search is coming out.

What is hierarchical search, as its name implies, is to query only a certain number of segments. When the result set is enough, the query will not continue, which is obvious for the optimization of search engine query performance.

However, there is a problem here, that is, the quality of the products released by the seller is uneven. We need to search out the products with good quality first. Therefore, the product quality of the previous segment is higher than that of the subsequent segment, otherwise, there will be no chance to present some high-quality exhibits. For example, if we have three segment, seg_1, seg_2, and seg_3, the product quality in seg_1 is higher than that in seg_2. The product quality in seg_2 is higher than that in seg_3, there are no requirements for each segment.

What are the criteria for determining product quality? We introduced a global score global_score. The global_score of each product is calculated offline and used as the basis for hierarchical search.

1. As shown in the offline computation of the search engine, there is a multi-table join link. In the multi-table join process, there will be some business logic computing, global_score is calculated at this stage. With global_score, we can sort products globally. If we generate three files after sorting, part_1, part_2, and part_3, The global_score of each doc in part_1 must be higher than that of every doc in part_2, and that of part_2 is also true for part_3, however, each part is not ordered internally. In the subsequent index creation process, there will be an ordering logic to ensure the order between multiple segments.

How does global sorting work? Because of the large amount of data, the offline computing tasks of each application basically run on the hadoop cluster, and the global sorting is also the case. To achieve the above effect, that is, the partitions are sorted by global_score. The solution we adopt is to first sample the data and partition by global_score, write the key of the defined partition to the _ partitions file, then implement the custom TotalOrderPartitioner (here the custom TotalOrderPartitioner is implemented to aggregate the products of the same company within a single output file, that is, by company_id aggregation, in this way, the compression ratio of output files is greatly increased, and the running time of the subsequent abuild index construction is significantly shortened) for global sorting. The core idea of sampling is to view only a small part of keys, obtain the approximate distribution of keys, and then build partitions.

It is necessary to first mention the concept of columns. Because a single search can carry a limited number of indexes, when the data volume is large, data needs to be divided so that all data is evenly distributed to different columns as much as possible. For example, if SC has 19 columns, all data is distributed to 19 columns Based on product_id % 19. After the join operation on multiple tables, the data is sorted. Therefore, global sorting is used to globally sort data in multiple columns.

When the layered search project is launched to the SC BT cluster (pre-release environment), Global sorting takes 80 minutes to complete. After analysis, most of the time is spent on sampling. After reading the code, we found that the global sorting of each column corresponds to a job. SC has 19 columns of data, and 19 jobs are run to sort the data of each column globally. Sampling is performed before sorting. The samplerger runs on the client. Therefore, it is especially important to limit the number of download parts to accelerate the operation of the samplerger. In the code implementation before optimization, each job reads data from the corresponding column and samples data independently. In addition, multiple jobs perform serial sampling. Therefore, a feasible optimization solution is to sample multiple jobs in parallel. However, because our product data is stored separately, the data volume in each column is large enough. For example, SC currently has 0.36 billion million data records, and the data in a single column is nearly million. Therefore, the global_score distribution of each product column is basically the same. Therefore, can we sample only one column of data and all jobs share this sample? In this way, not only the sampling time can be greatly shortened, but also the concurrency complexity will not be introduced. The answer is feasible.

Simply put, the basic idea of global sorting optimization is to share multiple global sorting jobs in multiple columns with the same sample based on the data distribution characteristics.

Next let's take a look at the optimized code implementation:

Vector vecRunningJob = new Vector(build_num);Vector vecJobClient = new Vector(build_num);for (int j = 0; j < build_num; j++) {    job.setJobName("Doc Sort job" + String.valueOf(j));    job.setInt("dc.sort.jobindex", j);    Vector vecInput = fileGenerator.getInPutFiles(j, build_num);    JobConf newjob = makeJob(job, inputPath, vecInput, outputPath + "/" + j, aggregateField); // Make a job for each column    JobClient jc = new JobClient(newjob);    vecJobClient.add(jc);    vecRunningJob.add(jc.submitJob(newjob));}

Build_num indicates the number of columns. From the code above, the makeJob method is called for each column of data, and then tasks are submitted for global sorting. Note that calling the makeJob method and submitting a task are serialized here, but the task is run in parallel after being submitted.

? Let's take a look at the implementation of the makeJob method:

private static JobConf makeJob(JobConf basejob, String inputPath,        Vector vecInPutFile, String outPutPath, String aggregateField) throws Exception {    JobConf conf = new JobConf(basejob);    conf.setJarByClass(DCSortMain.class);    for (int i = 0; i < vecInPutFile.size(); i++) {        FileInputFormat.addInputPath(conf, new Path(vecInPutFile.get(i)));    }    Path outputDir = new Path(outPutPath);    FileOutputFormat.setOutputPath(conf, outputDir);    conf.setMapOutputKeyClass(DCText.class);    conf.setMapOutputValueClass(DCText.class);    conf.setOutputKeyClass(DCText.class);    conf.setOutputValueClass(DCText.class);    conf.setMapperClass(IdentityMapper.class);    conf.setReducerClass(IdentityReducer.class);    conf.setInputFormat(DCTextInputFormat.class);    conf.setOutputFormat(DCTextOutputFormat.class);    conf.set("mapred.output.compress", "true");    conf.set("mapred.output.compression.codec", "org.apache.hadoop.io.compress.GzipCodec");    conf.setOutputKeyComparatorClass(DCText.AggregateFieldComparator.class); // Sort numbericly by desc    conf.setNumReduceTasks(conf.getInt("dc.sort.reduce_num", 1));    sample(conf, inputPath); // Sample before global sort    return conf;}

It can be seen that after relevant settings are completed, the makeJob will call the sample method for sampling. That is to say, the sample method will be called for each makeJob column.

Let's take a look at the implementation of the sample method:

private static void sample(JobConf conf, String inputPath) throws IOException, URISyntaxException {    int jobIndex = 0;    Path partitionFile = new Path(inputPath, jobIndex + "_partitions");    conf.setPartitionerClass(MyTotalOrderPartitioner.class);    conf.set("total.order.partitioner.natural.order", "false");    MyTotalOrderPartitioner.setPartitionFile(conf, partitionFile);    if (!sampleDone) {        LOG.info("sample start ...");        MyInputSampler.Sampler sampler =            new MyInputSampler.RandomSampler(1, 20000, 10);        MyInputSampler.writePartitionFile(conf, sampler);        LOG.info("sample end ...");        sampleDone = true;    }    // Add to DistributedCache    URI partitionUri = new URI(partitionFile.toString() + "#" + jobIndex + "_partitions");    DistributedCache.addCacheFile(partitionUri, conf);    DistributedCache.createSymlink(conf);} 

We can see that a Boolean variable sampleDone is introduced to control sampling. sampling is performed only when the makeJob method is called 1st times, and the jobs created later are not sampled, instead, it shares the same _ partitions file with 1st jobs and loads it into the distributed cache used by itself for global sorting. SampleDone is defined as follows:

private static boolean sampleDone = false; 

By the way, there are three built-in sampling tools for hadoop:

1) RandomSampler selects samples evenly from a dataset at the specified sampling rate;

2) SplitSampler: only samples the first n records in one shard;

3) IntervalSampler, which regularly selects keys from the Division at certain intervals, is a better choice for sorted data.

RandomSampler is an excellent general sampling device. We finally chose RandomSampler, because although the other two sampling devices are used, the sampling time is shorter, but the final data distribution is uneven. Only RandomSampler can achieve the expected results. At the same time, we set the sampling rate to 1, the maximum number of samples to 20000, and the maximum partition to 10. The maximum number of samples and the maximum partition only need to satisfy one requirement, that is, the sampling is stopped. You can adjust the parameters of RandomSampler to achieve different sampling effects.

After the optimized SC BT version is released, the running time of global sorting is reduced from 80 minutes to 30 minutes, and the time is shortened by 50 minutes. In the official environment, the hadoop cluster is more powerful and the running time of global sorting is shorter.

Original article address: Global sorting optimization, one of the optimization of international search offline systems. Thank you for sharing it with me.