My multithreaded ———— reads millions Excel data, writes to the database

Question: How to efficiently read millions Excel data to the database?

Problem teardown: 1. How do I read millions Excel data? There is a memory overflow in POI user mode.

2. Where is the read data stored in Excel? 1 million data in memory isn't that a scary thing?

3. How to write data to the database efficiently? How is multithreading used?

Scenario 1:poi command trigger mode to solve the memory overflow problem when reading, the data read into memory, in memory through the Forkjoin tool task disassembly, join to the queue

In-depth thinking: 1 "Every time you have to wait to read the data, in order to do other operations, read itself took 9 seconds time 2" 1 million data is very memory consumption AH

Scenario 2:2 Threads: Thread A and thread B, thread A is responsible for reading the Excel data, thread B is responsible for writing to the database, and how does thread A and thread B work together? Multi-threaded production consumer model

The final core code is attached:

POI Tool Class Code:

The read request uses the POI command mode, and the read is only responsible for adding data to the list, no limit

Core code for production consumption patterns

Use the thread's Factory mode to join the write request to the queue

Queue, constantly fetching write requests, executing

The final execution method of the write request

Warm tip: Database operations can use batch processing

My multithreaded ———— reads millions Excel data, writes to the database