Go concurrency sequencing

Stand-alone version external sort Arraysource

Support for chunking

Create a source for sorting by variable parameters ... The value of the passed-in setting, when returning the note to close (out)

Inmemsort

Create an internal sort function that gets and takes advantage of the sort from memory. Ints (a) to sort.

Use:

Merge

For multiple sources to sort, and the above difference is the input source one more, need to judge

Use

Readersource

Read the data source, Chunksize is to confirm the size of the read data, if you are not read after reading. Note Here the statement make ([]byte,8)

Writesink

Write Data

Randomsource

Generating data using random numbers

Mergen

Building a Merge node group

Merge sort

Use

External sorting consolidate small files

Use:

Specific implementation:

Run results (view size)

External sort consolidation large file added in pipeline0807

(Because the big files will read and play, we use the time to see what is going on)

Add output to Inmemsort and change out to 1024 when it is created

Add output to the merge

Added inside the external sort

Calling the init function in Createpipeline

Modify the main function: The computer is 4-core, so the parameter is 4 cost-effective, file size to 800000000 is slower, here I use the smaller

The output is roughly:

Go concurrency sequencing