Golang Aws-sdk-go's S3 service

This is a creation in Article, where the information may have evolved or changed.

S3 is a distributed file service offered by AWS, which is often used as a persistent storage for logs, input and output of large data processing results, etc.

S3 Service provides command-line tools, can easily upload, download, delete files, ordinary Golang program if you need to access files on S3, a simple way to download files on the S3 to local, and then directly access local files, but this method requires an additional step, download to local, There are additional operational costs, additional disk space is required, the use of the above is not very flexible, in addition, microservices should minimize the reliance on local data, this design does not conform to the design concept of microservices

Use Aws-sdk-go to directly access the S3 service for uploading and reading files

The following code is used: https://github.com/hatlonely/...

Create a session

First, you need to create a session, the subsequent access can be done through this session, if the access to the service requires authorization, you can also specify the authorization file in config

sess := session.Must(session.NewSession(&aws.Config{    Region: aws.String(endpoints.ApSoutheast1RegionID),}))service := s3.New(sess)

You must specify the region where the S3 bucket is located.

Uploading files

fp, err := os.Open("s3_test.go")So(err, ShouldBeNil)defer fp.Close()ctx, cancel := context.WithTimeout(context.Background(), time.Duration(30)*time.Second)defer cancel()_, err = service.PutObjectWithContext(ctx, &s3.PutObjectInput{    Bucket: aws.String("hatlonely"),    Key:    aws.String("test/s3_test.go"),    Body:   fp,})So(err, ShouldBeNil)

Using PutObjectWithContext the implementation of file upload, here can only implement file upload, can not implement file write, so can only write to local, and then the entire upload

Access time-outs can be set through the context

Download file

ctx, cancel := context.WithTimeout(context.Background(), time.Duration(30)*time.Second)defer cancel()out, err := service.GetObjectWithContext(ctx, &s3.GetObjectInput{    Bucket: aws.String("hatlonely"),    Key: aws.String("test/s3_test.go"),})So(err, ShouldBeNil)defer out.Body.Close()scanner := bufio.NewScanner(out.Body)for scanner.Scan() {    Println(scanner.Text())}

Using the GetObjectWithContext interface to read the file, the contents of the file are out. Body, you can use the scanner interface to continuously read the file contents by line

Finally remember to call out.Body.Close() and release resources

Traverse Directory

var objkeys []stringctx, cancel := context.WithTimeout(context.Background(), time.Duration(30)*time.Second)defer cancel()err := service.ListObjectsPagesWithContext(ctx, &s3.ListObjectsInput{    Bucket: aws.String("hatlonely"),    Prefix: aws.String("test/"),}, func(output *s3.ListObjectsOutput, b bool) bool {    for _, content := range output.Contents {        objkeys = append(objkeys, aws.StringValue(content.Key))    }    return true})So(err, ShouldBeNil)Println(objkeys)

Big data is usually the concurrent output, each node will output a file, to a specified directory, so sometimes we need to get a directory under the exactly what files, you can use to ListObjectsPagesWithContext traverse a directory of all the files, this function is recursive

Reference links

• AWS-SDK-GO:HTTPS://GITHUB.COM/AWS/AWS-SD ...
• Aws-sdk-go api:https://docs.aws.amazon.com/s ...

Reprint please indicate the source
This article link: http://hatlonely.github.io/20 ...