How to Use qiniu Python SDK to write a synchronization script and use the tutorial, pythonsdk

How to Use qiniu Python SDK to write a synchronization script and use the tutorial, pythonsdk

The Python SDK (Python-SDK) of qiniu cloud storage is an encapsulation of the qiniu cloud storage API Protocol, to provide a set of easy-to-use development tools for Python developers. Python developers do not need to understand the details of the qiniu cloud storage API Protocol when connecting to the Python-SDK. In principle, they do not need to have a deep understanding of the HTTP protocol and principle, however, if you have basic HTTP knowledge, you can process error scenarios more efficiently.

I recently set up a markdown static blog and want to put images in cloud storage.

After research, I found that qiniu could meet my individual needs and I chose it.

To reference an image, you must first upload the image to the cloud.

Although the qianniu website can upload files in the background, you must log on to the backend of each upload, select an image, and set the connection address to upload files.

This process is complicated, so I want to use the SDK provided by qiniu cloud to write a synchronization tool to facilitate incremental file synchronization.

With this idea, we will act immediately. It took about one morning to write the tool and put it on GitOSC and github.

#! /Usr/bin/env python #-*-coding: UTF-8-*-# AUTHOR = "heqingpan" # AUTHOR_EMAIL = "heqingpan@126.com" # URL = "http://git.oschina.net/hqp/qiniu_sync" import qiniufrom qiniu import Authfrom qiniu import BucketManagerimport osimport reaccess_key = ''secret _ key = ''bucket _ name = ''bucket _ domain ='' q = Auth (access_key, secret_key) bucket = BucketManager (q) basedir = OS. path. realpath (OS. path. dirname (_ file _) filename =__ file _ ignore_paths = [filename, "{0} c ". format (filename)] ignore_names = [". DS_Store ",". git ",". gitignore "] charset =" utf8 "diff_time = 2 * 60def list_all (bucket_name, bucket = None, prefix =" ", limit = 100): rlist = [] if bucket is None: bucket = BucketManager (q) marker = None eof = False while eof is False: ret, eof, info = bucket. list (bucket_name, prefix = prefix, marker = marker, limit = limit) marker = ret. get ('marker', None) for item in ret ['items ']: rlist. append (item ["key"]) if eof is not True: # error handling # print "error" pass return rlistdef get_files (basedir = "", fix = "", rlist = None, ignore_paths = [], ignore_names = []): if rlist is None: rlist = [] for subfile in OS. listdir (basedir): temp_path = OS. path. join (basedir, subfile) tp = OS. path. join (fix, subfile) if tp in ignore_names: continue if tp in ignore_paths: continue if OS. path. isfile (temp_path): rlist. append (tp) elif OS. path. isdir (temp_path): get_files (temp_path, tp, rlist, ignore_paths, ignore_names) return rlistdef values (subdir = ""): basedir = subdir or basedir files = get_files (basedir = basedir, ignore_paths = ignore_paths, ignore_names = ignore_names) return map (lambda f :( f. replace ("\", "/"), f), files) def sync (): qn_keys = list_all (bucket_name, bucket) qn_set = set (qn_keys) l_key_files = get_valid_key_files (basedir) k2f = {} update_keys = [] u_count = 500 u_index = 0 for k, f in l_key_files: k2f [k] = f str_k = k if isinstance (k, str): k = k. decode (charset) if k in qn_set: update_keys.append (str_k) u_index + = 1 if u_index> u_count: u_index-= u_count update_file (k2f, update_keys) update_keys = [] else: # upload upload_file (k, OS. path. join (basedir, f) if update_keys: update_file (k2f, update_keys) print "sync end" def update_file (k2f, ulist): ops = qiniu. build_batch_stat (bucket_name, ulist) rets, infos = bucket. batch (ops) for I in xrange (len (ulist): k = ulist [I] f = k2f. get (k) ret = rets [I] ["data"] size = ret. get ("fsize", None) put_time = int (ret. get ("putTime")/10000000) local_size = OS. path. getsize (f) local_time = int (OS. path. getatime (f) if local_size = size: continue if put_time> = local_time-diff_time: # is new continue # update upload_file (k, OS. path. join (basedir, f) def upload_file (key, localfile): print "upload_file:" print key token = q. upload_token (bucket_name, key) mime_type = get_mime_type (localfile) params = {'x: a': 'A'} progress_handler = lambda progress, total: progress ret, info = qiniu. put_file (token, key, localfile, params, mime_type, progress_handler = progress_handler) def get_mime_type (path): mime_type = "text/plain" return mime_typedef main (): sync () if _ name __= = "_ main _": main ()

This synchronization script supports batch file comparison, incremental differential update, and batch update.

Usage

Install qiniu Python SDK

pip install qiniu

Fill in the configuration information of the script file (qiniusync. py)

access_key = ''secret_key = ''bucket_name = ''

You can get the corresponding information after registration.

Copy the script file (qiniusync. py) to the root directory to be synchronized.

Run scripts

python qiniusync.py

Postscript

After writing the submission, I found that qiniu has provided the corresponding tools. I have already made the wheel.

Since it has been written, it will be released. It may be useful to be familiar with the sdks of qiniu.

Qiniu cloud storage Python SDK usage tutorial

This tutorial describes how to use the Python SDK of qiniu to quickly upload, download, process, and manage files.

Install

First, you must install Python SDK first. Seven cattle Python SDK is open-source, hosted on Github, the Project address is https://github.com/qiniu/python-sdk.

You can use pip install qiniu as described in the project description. Of course, you can directly clone the source code and use it directly. I generally like to clone the source code directly. In this way, it is very easy to make some changes to the SDK.

The latest version of Python SDK depends on the requests library, so you must install it in advance. You can also use pip install requests for installation.

Development Environment

Python has many development environments. If you like text, such as vim, emacs, and sublime text, it is a good choice. If you like IDE, PyCharm is the most popular. The latest version of PyCharm is downloaded here.

Access Key and Secret Key

We know that the permission Verification Mechanism of qiniu cloud storage is based on a pair of keys called Access Key and Secret Key. The Access Key is the public Key and the Secret Key is the private Key. This pair of keys can be obtained from the backend of qiniu.

Test Tool

Well, after completing the above preparations, we will upload a simple file and train our hands.

python#coding=utf-8__author__ = 'jemy''''

This example demonstrates a simple file upload.

In this example, the sdk selects Form upload or multipart Upload Based on the file size.

'''Import qiniuaccessKey = "<Your Access Key>" secretKey = "<Your Secret Key>" # resolution result def parseRet (retData, respInfo): if retData! = None: print ("Upload file success! ") Print (" Hash: "+ retData [" hash "]) print (" Key: "+ retData [" key "]) # Check the extension parameter for k, v in retData. items (): if k [: 2] = "x:": print (k + ":" + v) # Check other parameters for k, v in retData. items (): if k [: 2] = "x:" or k = "hash" or k = "key": continue else: print (k + ": "+ str (v) else: print (" Upload file failed! ") Print (" Error: "+ respInfo. text_body) # upload without a key. The key parameter def upload_without_key (bucket, filePath) is not specified in the http request: # generate the upload credential auth = qiniu. auth (accessKey, secretKey) upToken = auth. upload_token (bucket, key = None) # upload the file retData, respInfo = qiniu. put_file (upToken, None, filePath) # parse the result parseRet (retData, respInfo) def main (): bucket = "if-pbl" filePath = "/Users/jemy/Documents/jemy.png" upload_without_key (bucket, filePath) if _ name _ = "_ main _": main ()

The running result is:

Upload file success!
Hash: Fp0XR6tM4yZmeiKXw7eZzmeyYsq8
Key: Fp0XR6tM4yZmeiKXw7eZzmeyYsq8

From the above we can see that the most basic steps for uploading files using the Python SDK of qiniu are:

1. generate an upload credential

2. upload files

3. parse the reply result

Summary

To sum up, it is very easy to use the SDK of qiniu to upload files. In the next tutorial, we will gradually learn about it based on this example.