大資料拆分處理方式

來源:互聯網
上載者:User

標籤:

思路:
1:參考分頁方式將資料拆成指定大小線程數
2:在每個獨立的線程中去讀取資料並處理資料
步驟一實現

public class IndexIntiTools {        public static AtomicInteger runflag=new AtomicInteger();    //用於測試    public static List<Object>  syncList=new CopyOnWriteArrayList<Object>();    private static int idxThreadCount = 10;    private static Executor ex = Executors.newFixedThreadPool(idxThreadCount);    /**     * 構建索引     * @param hql     * @param size     */    public static void build(String hql, int size) {        int pagecount = idxThreadCount;        int count = size / pagecount;        int mod = size % pagecount;        List<Runnable> runList = new ArrayList<Runnable>(pagecount);        IndexExecutor idxExecutor;        for (int i = 0; i < pagecount; i++) {            if (i == (pagecount - 1)) {                idxExecutor = new IndexExecutor(hql, i * count, count + mod);            } else {                idxExecutor = new IndexExecutor(hql, i * count, count);            }            runList.add(idxExecutor);        }        for (Runnable runnable : runList) {            runflag.incrementAndGet();            ex.execute(runnable);        }    }}

步驟二實現

public class IndexExecutor implements Runnable{        private static final Log log = LogFactory.getLog(IndexExecutor.class);    private int start;    private int limit;    private String hql;    public IndexExecutor(String hql,int start, int limit) {        this.hql=hql;        this.start = start;        this.limit = limit;    }    @Override    public void run() {        log.info("hql:"+hql+",start:"+start+",limit"+limit);        //查詢資料庫(hql,start,limit);        log.info(list);        IndexIntiTools.syncList.addAll(list);        IndexIntiTools.runflag.decrementAndGet();    }}

大資料拆分處理方式

相關文章

聯繫我們

該頁面正文內容均來源於網絡整理,並不代表阿里雲官方的觀點,該頁面所提到的產品和服務也與阿里云無關,如果該頁面內容對您造成了困擾,歡迎寫郵件給我們,收到郵件我們將在5個工作日內處理。

如果您發現本社區中有涉嫌抄襲的內容,歡迎發送郵件至: info-contact@alibabacloud.com 進行舉報並提供相關證據,工作人員會在 5 個工作天內聯絡您,一經查實,本站將立刻刪除涉嫌侵權內容。

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.