Uploading files from a Windows EC2 instance to a S3 bucket using Python Boto3

First, create the end node

Why create endpoints and manage your vpc and S3? If you do not manage the VPC and S3 through the endpoint, the EC2 instance Access S3 bucket in the VPC is through the public network, and once associated, the EC2 instance in the VPC accesses the S3 bucket to the internal network. The benefits are two: 1. The internal network will not incur traffic costs; 2. Walking the internal network is fast and will not cause our Python script to produce an exception for network reasons.

Vpc-> endpoint, create endpoint, associate VPC and S3, associated subnets

Ii. installing the Python3 compiler and the Boto3 library in Windows

1.: https://www.python.org/

2. Double-click Install, default installation path "C:\Users\ user \appdata\local\programs\python\python36"

3. Configure Environment variables

4. Install the Boto3 Development Library (the environment variable is ready to use the PIP command)

Third, generate AWS IAM user keys and configure

1. iam-> user, select User with Access S3 permissions, security certificate, create Access security key, download key file to local

2. Configure AWS key authentication on a Windows instance

A) Create the ~/.aws/credentials file with the following file contents: [default]aws_access_key_id = Xxxxxxaws_secret_access_key = xxxxxxb) Create ~/.aws/config File, the contents of the file are as follows: [default]region=cn-north-1

Third, edit the Python3 script, the foot name is "s3_upload.py"

Import osimport datetimeimport boto3import loggingfrom boto3.s3.transfer import  transferconfiglogging.basicconfig (level=logging.info,                 format= '% (asctime) s % (filename) s[line:% (Lineno) d] % ( LevelName) s % (message) s ',                 datefmt= '%a, %d %b %y %h:%m:%s ',                 filename= ' E:\\xxx\\xxx\\xxx\\aws_upload.log ',                 filemode= ' a ') Delta = datetime.timedelta (days=2) now = datetime.datetime.now () s3 = boto3.client (' S3 ') bucket_name =  ' daily-backup ' file_dir= ' e:\\xxx\\xxx\\xxx ' gb = 1024 ** 3#   A single file larger than 10GB, you need to set this value config = transferconfig (MULTIPART_THRESHOLD=5&NBSP;*&NBSP;GB) os.chdir (file_dir) file_list =  Os.listdir () for file in file_list:    #  upload only zip files      if file.endswith ('. zip '):        #  uploaded two days ago files generated          ctime = datetime.datetime.fromtimestamp (Os.path.getctime ( File)         if ctime <  (Now-delta):             try:                 s3.upload_file (file, bucket_name, file,  Config=config)             except exception  as e:                 logging.error (e)   &nbsP;             logging.error ("%s upload  failed. "  % file)             else:                 #  upload succeeds delete local file                  logging.info ("%s  upload successful. "  % file)                  os.remove (file)

Iv. testing and scheduling of scheduled tasks

1. Manually run the Python script you just edited on the Windows cmd command line

2. If successful, edit the Windows timed task to upload files from the local directory to the S3 bucket on a daily schedule

V. Setting the S3 bucket life cycle

For files uploaded to a S3 bucket, we want to delete files from the previous 30 days regularly, and we can set the bucket life cycle and automatically delete outdated files.

Add life cycle Rules

Uploading files from a Windows EC2 instance to a S3 bucket using Python Boto3