Sample a balance dataset from imbalance dataset and save it（從不平衡資料中抽取平衡資料，並儲存）

標籤：classes   roc   close   imp   儲存   value   random   subclass   collect   

 

有時我們在實際分類資料採礦中經常會遇到，類別樣本很不均衡，直接使用這種不均衡資料會影響一些模型的分類效果，如logistic regression，SVM等，一種解決辦法就是對資料進行均衡採樣，這裡就提供了一個建議代碼實現，要求輸入和輸出資料格式為Label+Tab+Features, 如Libsvm format

-1 1:0.875 2:-1 3:-0.333333 4:-0.509434 5:-0.347032 6:-1 7:1 8:-0.236641 9:1 10:-0.935484 11:-1 12:-0.333333 13:-1 +1 1:0.166667 2:1 3:-0.333333 4:-0.433962 5:-0.383562 6:-1 7:-1 8:0.0687023 9:-1 10:-0.903226 11:-1 12:-1 13:1 +1 1:0.708333 2:1 3:1 4:-0.320755 5:-0.105023 6:-1 7:1 8:-0.419847 9:-1 10:-0.225806 12:1 13:-1 -1 1:0.583333 2:-1 3:0.333333 4:-0.603774 5:1 6:-1 7:1 8:0.358779 9:-1 10:-0.483871 12:-1 13:1 

 

用法 Usage：

Usage: {0} [options] dataset subclass_size [output]options:-s method : method of selection (default 0)     0 -- over-sampling & under-sampling given subclass_size     1 -- over-sampling (subclass_size: any value)     2 -- under-sampling(subclass_size: any value)

 Bash example：

python SampleDataset.py -s 0 heart_scale 20 heart_scale.txt

這裡s參數表示抽樣的方法，

-s 0：Over sampling &Under sampling ，即對類別多的進行降採樣，對類別少的進行重採樣

-s 1: Over sampling 對類別少的進行重採樣，採樣後的每類樣本數與最多的那一類一致

-s 2：Under sampling 對類別多的進行降採樣，採樣後的每類樣本數與最少的那一類一值 

輸入資料檔案heart_scale

輸出資料檔案heart_scale.txt

 

下面是代碼檔案：SampleDataset.py：

#!/usr/bin/env pythonfrom sklearn.datasets import load_svmlight_filefrom sklearn.datasets import dump_svmlight_fileimport numpy as npfrom sklearn.utils import check_random_statefrom scipy.sparse import hstack,vstackimport os, sys, math, randomfrom collections import defaultdictif sys.version_info[0] >= 3:    xrange = rangedef exit_with_help(argv):    print("""Usage: {0} [options] dataset subclass_size [output]options:-s method : method of selection (default 0)     0 -- over-sampling & under-sampling given subclass_size     1 -- over-sampling (subclass_size: any value)     2 -- under-sampling(subclass_size: any value)output : balance set file (optional)If output is omitted, the subset will be printed on the screen.""".format(argv[0]))    exit(1)def process_options(argv):    argc = len(argv)    if argc < 3:        exit_with_help(argv)    # default method is over-sampling & under-sampling    method = 0      BalanceSet_file = sys.stdout    i = 1    while i < argc:        if argv[i][0] != "-":            break        if argv[i] == "-s":            i = i + 1            method = int(argv[i])            if method not in [0,1,2]:                print("Unknown selection method {0}".format(method))                exit_with_help(argv)        i = i + 1    dataset = argv[i]      BalanceSet_size = int(argv[i+1])    if i+2 < argc:        BalanceSet_file = open(argv[i+2],‘w‘)    return dataset, BalanceSet_size, method, BalanceSet_filedef stratified_selection(dataset, subset_size, method):    labels = [line.split(None,1)[0] for line in open(dataset)]    label_linenums = defaultdict(list)    for i, label in enumerate(labels):        label_linenums[label] += [i]    l = len(labels)    remaining = subset_size    ret = []    # classes with fewer data are sampled first;     label_list = sorted(label_linenums, key=lambda x: len(label_linenums[x]))    min_class = label_list[0]    maj_class = label_list[-1]    min_class_num = len(label_linenums[min_class])    maj_class_num = len(label_linenums[maj_class])    random_state = check_random_state(42)    for label in label_list:        linenums = label_linenums[label]        label_size = len(linenums)        if  method == 0:            if label_size<subset_size:                ret += linenums                subnum = subset_size-label_size            else:                subnum = subset_size            ret += [linenums[i] for i in random_state.randint(low=0, high=label_size,size=subnum)]        elif method == 1:            if label == maj_class:                ret += linenums                continue            else:                ret += linenums                subnum = maj_class_num-label_size                                ret += [linenums[i] for i in random_state.randint(low=0, high=label_size,size=subnum)]        elif method == 2:            if label == min_class:                ret += linenums                continue            else:                subnum = min_class_num                ret += [linenums[i] for i in random_state.randint(low=0, high=label_size,size=subnum)]    random.shuffle(ret)    return retdef main(argv=sys.argv):    dataset, subset_size, method, subset_file = process_options(argv)    selected_lines = []    selected_lines = stratified_selection(dataset, subset_size,method)    #select instances based on selected_lines    dataset = open(dataset,‘r‘)    datalist = dataset.readlines()    for i in selected_lines:        subset_file.write(datalist[i])    subset_file.close()    dataset.close()if __name__ == ‘__main__‘:    main(sys.argv)

 

Sample a balance dataset from imbalance dataset and save it（從不平衡資料中抽取平衡資料，並儲存）