Build independent image service based on Minio and Thumbor

This is a creation in Article, where the information may have evolved or changed.

Most applications use image storage, and from a system architecture perspective, services such as image storage should be stripped out of the core business as much as possible. Many people will choose online cloud storage services, such as seven of cow cloud storage. However, many enterprise projects require image services to be deployed internally because of various requirements. So we may still need a picture service that can be deployed independently.

Self-development to achieve a set of image storage services system, it will take a lot of effort, but if there are ready-made solutions to do? I found on GitHub that the two projects of Minio and Thumbor can help us achieve our needs.

Minio

Minio Cloud Storage is a distributed object storage System. It is an open source project based on go and implements the Amazon S3 API. This means that Minio is equivalent to an open source Amazon S3.

Using a Docker installation can be the easiest way, with two lines of commands to install and download and run. Note that this startup mode is limited to the test environment and the data will disappear once the command is terminated. For installation deployment in a production environment, refer to the documentation.

docker pull minio/miniodocker run -p 9000:9000 minio/minio server /export

After startup, a heap of system information is displayed on the command line.

Endpoint:  http://172.17.0.2:9000  http://127.0.0.1:9000AccessKey: PT7TIDXEV7KH7S5R91JHSecretKey: 17hdYH0mF2Ljd3ahZ7JGyi5l/ZPjunLlldUMY/A0Region:    us-east-1SQS ARNs:  <none>

You can then access it in the browser http://127.0.0.1:9000 and log in using accesskey and Secretkey.

Its core function is simple, create buckets, upload files. All of its file links are private links that contain expiration times. Then we can use the SDK that it provides, which is just as simple.

Minio is focused on file storage and does not have any image processing capabilities, but we can use the Thumbor service to do image processing.

Thumbor

Thumbor is a very powerful image processing service that enables image cropping, zooming, filters, and even face recognition.

It is a python-based open source project that can be installed in a Python environment via PIP.

pip install thumbor

You can then run it using the command below.

The service runs locally on port 8888 and can then be invoked directly through the URL.

Like this picture.https://www.apple.com/cn/home/images/gallery/iphone_alt_small_2x.jpg

You can use http://localhost:8888/unsafe/300x200/https://www.apple.com/cn/home/images/gallery/iphone_alt_small_2x.jpg this URL to zoom to the 300x200 size.

Thumbor AWS

Thumbor processing pictures is very powerful, but only provides a very weak picture storage function. The image upload modification interface is not verified, so it is not turned on by default. But Thumbor has some community support, and third-party developers do some extensions. The Thumbor AWS extension can integrate Thumbor backend with Amazon S3. The back end of the Key,thumbor that submits the picture file on the URL will take the file out of Amazon S3 for processing. Previously, Minio was compatible with Amazon S3 APIs. So it means that only a little change is needed to integrate the two services. Finally, you can achieve, using Minio upload storage images, using Thumbor to take out pictures to do processing.

We use pip it to install this extension.

pip install tc_aws

Thumbor AWS This extension leverages Boto3 to connect to Amazon S3 's SDK, and according to the configuration in the Boto3 documentation, we need to create this file and fill in the Minio ~/.aws/credentials and accesskey that just secretkey provided, Authorize this service to access Minio.

[default]aws_access_key_id = YOUR_ACCESS_KEYaws_secret_access_key = YOUR_SECRET_KEY

Some of the configurations are then modified using the thumbor_conf generated thumbor.conf.

TC_AWS_REGION='us-east-1' #填写minio提供的RegionTC_AWS_ENDPOINT='http://127.0.0.1:9000' #将默认的Amazon S3地址换成minio运行地址LOADER = 'tc_aws.loaders.s3_loader' #将Thumbor的loader换成tc_aws.loaders.s3_loader

Restart Thumbor, and then you can access the pictures in the Minio through Thumbor in your browser.

For example, a aaa.jpg is uploaded in the bucket of Minio Docker, and then we can access the http://localhost:8888/unsafe/docker/aaa.jpg original image from this address. Of course, we can http://localhost:8888/unsafe/300x200/docker/aaa.jpg 300x200 this image in this way.

So we're done.

Summarize

Each service has its own fine side, but fortunately Amazon S3 has a de facto standard at the Cloud service API level, so this can help us integrate some of the services together.

As a result, the image storage and extraction scaling requirements are finally achieved, and are sufficient for some projects. However, after the image is processed by Thumbor, the properties of its private link are gone. This is also awkward in some demanding corporate services projects.

A simple look at the next Thumbor source, its picture output is tornado realized, I believe there is time to hack source, the realization of private links will not be too difficult.

Enjoy it!