Multiple filtering of large amounts of data in JavaScript

Data filtering is not complicated. You only need to clarify your ideas and understand what data needs to be retained and what data is temporary (intermediate process, what data is the final result ...... Using the related methods in Array. prototype, or tools such as lodash, it is easy to process them. All codes use ES2015 syntax. If ES5 syntax is required, you can use Babel-Try it out or TypeScript Playground for translation.

Problem proposal

Today, a friend asked me a question. The front-end obtains a large amount of data from the backend through Ajax and needs to be filtered based on some conditions. The filtering method is as follows:

class Filter {     filterA(s) {         let data = this.filterData || this.data;         this.filterData = data.filter(m => m.a === s);     }          filterB(s) {         let data = this.filterData || this.data;         this.filterData = data.filter(m => m.b === s);     } }

I am confused and feel that this process is not correct, but I don't know how to handle it.

Problems Found

The problem lies in filtering. This means that you can implement multiple filters (call filelist () and then call filterB (), but this filter is irreversible. If the filtering process is like this:

f.filterA("a1"); f.filterB("b1"); f.filterA("a2");

If you want to filter data by "a1" and "b1", then modify the first condition to "a2", but the result becomes an empty set.

Solve the problem

If problems are found, they are solved accordingly. This problem occurs because the filtering process is irreversible. It can be solved by directly filtering from this. data each time, instead of filtering from this. filterData. To do this, you need to record the selected filter conditions first.

Record filtering Conditions

It is feasible to use a list record filter condition, but note that the two filters for the same condition are mutually exclusive and only the last one can be retained. Therefore, HashMap should be more suitable.

class Filter {     constructor() {         this.filters = {};     }      set(key, filter) {         this.filters[key] = filter;     }      getFilters() {         return Object.keys(this.filters).map(key => this.filters[key]);     } }

In this case, the process shown above is

f.set("A", m => m.a === "a1"); f.set("B", m => m.b === "b1"); f.set("A", m => m.a === "a1"); let filters = f.getFilters(); // length === 2;

The filter set in the above 3rd sentences overwrites the filter set in the 1st clause. Now we can use the final filters to filter the original data this. data in sequence to get the correct results.

Some people will think that the list returned by getFilters () is not in the set order -- indeed, this is a feature of HashMap and is unordered. However, the results are the same no matter who is the first. However, some compound conditions may be affected.

If necessary, we can use array instead of map to solve the order problem, but this will reduce the search efficiency (linear search ). If you want to solve the search efficiency problem, you can use array + map to handle it. I will not talk about it here.

Filter

In fact, when using getFilter (), it is really slow to use a loop to process it every time. Since data is encapsulated into a Filter, you can consider directly sending a filter () method to the Filter interface.

class Filter {     filter() {         let data = this.data;         for (let f of this.getFilters()) {             data = data.filter(f);         }         return data;     } }

However, I think the efficiency is not very good, especially for a large amount of data. Use the lodash delay processing process.

Lodash-based latency Processing

filter() {     let chain = _(this.data);     for (let f of this.getFilters()) {         chain = chain.filter(f);     }     return chain.value(); }

Lodash enables the delay processing when the data is greater than 200. That is to say, it calls each filter in a loop, instead of repeating each filter.

The difference between latency processing and non-latency processing is obvious. A total of n (n = 3) large loops are performed for non-delayed processing to generate n-1 intermediate results. However, latency processing only performs a large loop without intermediate results.

But to be honest, I don't like loading a library for a few things, so I just want to make a simple implementation myself.

Implement latency processing by yourself

Filter () {const filters = this. getFilters (); return data. filter (m => {for (let f of filters) {// if a filter has filtered it out, you do not need to use the filter to determine if (! F (m) {return false ;}} return true ;});}

The for loop can also be simplified using Array. prototype. every:

filter() {     const filters = this.getFilters();     return data.filter(m => {         return filters.every(f => f(m));     }); }

Data filtering is not complicated. You only need to clarify your ideas and understand what data needs to be retained and what data is temporary (intermediate process, what data is the final result ...... Using the related methods in Array. prototype, or tools such as lodash, it is easy to process them.