caffe 教程 Fine-tuning CaffeNet for Style Recognition on “Flickr Style” Data 遇到的問題及解決方案

在使用這個教程時，主要遇到了兩個問題：

1、資料下不下來。

python examples/finetune_flickr_style/assemble_data.py --workers=1 --images=2000 --seed 831486

運行上述指令時，程式莫名其妙就不動了，也不下載檔案，程式也沒有掛掉，好像進入了死結狀態。

查看來源程式：assemble_data.py，可以看出assemble_data.py用了大量多線程，多進程。我的解決方案就是改來源程式，不使用進程來下載了。並且，對下載進行了逾時限定，超過6s就認為逾時，進而不下載。

====================================================================================================

assemble_data.py中使用多線程，多進程的原始碼如下：

    pool = multiprocessing.Pool(processes=num_workers)    map_args = zip(df['image_url'], df['image_filename'])    results = pool.map(download_image, map_args)

===================================================================================================

我修改後的源碼如下：

#!/usr/bin/env python3"""Form a subset of the Flickr Style data, download images to dirname, and writeCaffe ImagesDataLayer training file."""import osimport urllibimport hashlibimport argparseimport numpy as npimport pandas as pdfrom skimage import ioimport multiprocessingimport socket# Flickr returns a special image if the request is unavailable.MISSING_IMAGE_SHA1 = '6a92790b1c2a301c6e7ddef645dca1f53ea97ac2'example_dirname = os.path.abspath(os.path.dirname(__file__))caffe_dirname = os.path.abspath(os.path.join(example_dirname, '../..'))training_dirname = os.path.join(caffe_dirname, 'data/flickr_style')def download_image(args_tuple):    "For use with multiprocessing map. Returns filename on fail."    try:        url, filename = args_tuple        if not os.path.exists(filename):            urllib.urlretrieve(url, filename)        with open(filename) as f:            assert hashlib.sha1(f.read()).hexdigest() != MISSING_IMAGE_SHA1        test_read_image = io.imread(filename)        return True    except KeyboardInterrupt:        raise Exception()  # multiprocessing doesn't catch keyboard exceptions    except:        return Falsedef mydownload_image(args_tuple):    "For use with multiprocessing map. Returns filename on fail."    try:        url, filename = args_tuple        if not os.path.exists(filename):            urllib.urlretrieve(url, filename)        with open(filename) as f:            assert hashlib.sha1(f.read()).hexdigest() != MISSING_IMAGE_SHA1        test_read_image = io.imread(filename)        return True    except KeyboardInterrupt:        raise Exception()  # multiprocessing doesn't catch keyboard exceptions    except:        return Falseif __name__ == '__main__':    parser = argparse.ArgumentParser(        description='Download a subset of Flickr Style to a directory')    parser.add_argument(        '-s', '--seed', type=int, default=0,        help="random seed")    parser.add_argument(        '-i', '--images', type=int, default=-1,        help="number of images to use (-1 for all [default])",    )    parser.add_argument(        '-w', '--workers', type=int, default=-1,        help="num workers used to download images. -x uses (all - x) cores [-1 default]."    )    parser.add_argument(        '-l', '--labels', type=int, default=0,        help="if set to a positive value, only sample images from the first number of labels."    )    args = parser.parse_args()    np.random.seed(args.seed)    # Read data, shuffle order, and subsample.    csv_filename = os.path.join(example_dirname, 'flickr_style.csv.gz')    df = pd.read_csv(csv_filename, index_col=0, compression='gzip')    df = df.iloc[np.random.permutation(df.shape[0])]    if args.labels > 0:        df = df.loc[df['label'] < args.labels]    if args.images > 0 and args.images < df.shape[0]:        df = df.iloc[:args.images]    # Make directory for images and get local filenames.    if training_dirname is None:        training_dirname = os.path.join(caffe_dirname, 'data/flickr_style')    images_dirname = os.path.join(training_dirname, 'images')    if not os.path.exists(images_dirname):        os.makedirs(images_dirname)    df['image_filename'] = [        os.path.join(images_dirname, _.split('/')[-1]) for _ in df['image_url']    ]    # Download images.    num_workers = args.workers    if num_workers <= 0:        num_workers = multiprocessing.cpu_count() + num_workers    print('Downloading {} images with {} workers...'.format(        df.shape[0], num_workers))    #pool = multiprocessing.Pool(processes=num_workers)    map_args = zip(df['image_url'], df['image_filename'])    #results = pool.map(download_image, map_args)    socket.setdefaulttimeout(6)    results = []    for item in map_args:        value = mydownload_image(item)        results.append(value)        if value == False:                print 'Flase'        else:                print '1'    # Only keep rows with valid images, and write out training file lists.    print len(results)    df = df[results]    for split in ['train', 'test']:        split_df = df[df['_split'] == split]        filename = os.path.join(training_dirname, '{}.txt'.format(split))        split_df[['image_filename', 'label']].to_csv(            filename, sep=' ', header=None, index=None)    print('Writing train/val for {} successfully downloaded images.'.format(        df.shape[0]))

修改主要有以下幾點：

1、#!/usr/bin/env python3 使用python3

2、

#pool = multiprocessing.Pool(processes=num_workers)    map_args = zip(df['image_url'], df['image_filename'])    #results = pool.map(download_image, map_args)    socket.setdefaulttimeout(6)    results = []    for item in map_args:        value = mydownload_image(item)        results.append(value)        if value == False:                print 'Flase'        else:                print '1'    # Only keep rows with valid images, and write out training file lists.    print len(results)

只使用單線程下載，不使用多線程，多進程下載。並且，設定串連的逾時時間為6s,socket.setdefaulttimeout(6)。

經過上述改進，就可以把資料下載下來。

===================================================================================================

2、

在運行命令： 

./build/tools/caffe train -solver models/finetune_flickr_style/solver.prototxt -weights models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel

時遇到錯誤：

Failed to parse NetParameter file: models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel

出錯的原因是我們傳入的資料bvlc_reference_caffenet.caffemodel 並不是二進位的。

原因：因為我是在win7下，把bvlc_reference_caffenet.caffemodel下載下來，再使用winSCP傳輸到伺服器上，直接在伺服器上使用wget下載，速度太慢了，但是在傳輸的過程中winSCP就把bvlc_reference_caffenet.caffemodel的格式給篡改了,導致bvlc_reference_caffenet.caffemodel不是二進位的。

解決方案，把winSCP的傳輸格式設定成二進位，那麼就可以解決這個問題。

詳情見部落格:http://blog.chinaunix.net/uid-20332519-id-5585964.html