How to Create a cloud-based Encrypted File System in Linux

How to Create a cloud-based Encrypted File System in Linux

Commercial Cloud Storage services such as Amazon S3 and Google Cloud Storage provide high availability, scalability, and unlimited capacity object Storage services at an affordable price. To accelerate the widespread adoption of these cloud products, these providers have developed a good developer ecosystem for their products through clear APIs and sdks. The cloud-based file system is a typical product in these active developer communities and has several open-source implementations.

S3QL is one of the most popular open-source cloud file systems. It is a FUSE-based file system that provides several commercial or open-source Cloud Storage backends, such as Amazon S3, Google Cloud Storage, Rackspace CloudFiles, and OpenStack. As a fully functional file system, S3QL has many powerful functions: file Size, compression, UNIX properties, encryption, snapshot Based on copy at write time, immutable tree, deduplication, and support for soft and hard links. All data written to the S3QL file system will be locally compressed and encrypted before being transmitted to the cloud backend. When you try to retrieve content from the S3QL file system, if they are not in the local cache, the corresponding objects will be downloaded from the cloud, and then decrypted and decompressed instantly.

It must be clear that S3QL does have its limitations. For example, you cannot mount the same S3FS File System on several different computers at the same time, but only one computer can access it at the same time. In addition, the ACL (Access Control List) is not supported.

In this tutorial, I will describe "how to configure an encrypted file system with S3QL Based on Amazon S3 ". As an example, I will also explain how to run the rsync backup tool on the mounted S3QL file system.

Preparations

In this tutorial, you must first create an Amazon AWS account (registration is free, but a valid credit card is required ).

Create an AWS access key (access key ID and secret access key). S3QL uses this information to access your AWS account.

Then, access AWS S3 through the AWS Management Panel and create a new empty bucket for S3QL.

For optimal performance, select a region that is geographically closest to you.

Install S3QL on Linux

In most Linux distributions, pre-compiled S3QL packages are available.

For Debian, Ubuntu, or Linux Mint:

1. $ Sudo apt-get install s3ql

For Fedora:

1. $ Sudo yum install s3ql

For Arch Linux, use AUR.

Configure S3QL for the first time

In ~ /. Create the autoinfo2 file in the s3ql Directory, which is a default configuration file of S3QL. The information in this file includes the required AWS access key, S3 bucket name, and encrypted password. This encryption password will be used to encrypt a randomly generated CMK, And the CMK will be used to actually encrypt S3QL file system data.

1. $ Mkdir ~ /. S3ql
2. $ Vi ~ /. S3ql/authinfo2

1. [S3]
2. Storage-url: s3: // [bucket-name]
3. Backend-login: [your-access-key-id]
4. Backend-password: [your-secret-access-key]
5. Fs-passphrase: [your-encryption-passphrase]

The specified AWS S3 bucket needs to be created through the AWS Management Panel in advance.

For security, make the authinfo2 file accessible only to you.

1. $ Chmod 600 ~ /. S3ql/authinfo2

Create an S3QL File System

Now you are ready to create an S3QL File System on AWS S3.

Use the mkfs. s3ql tool to create a new S3QL file system. The bucket name in this command should be consistent with the name specified in the authinfo2 file. The "-- ssl" parameter is used to forcibly connect to the backend storage server. By default, The mkfs. s3ql command enables compression and encryption in the S3QL file system.

1. $ Mkfs. s3ql s3: // [bucket-name] -- ssl

You will be asked to enter an encrypted password. Enter ~ The password specified by fs-passphrase in/. s3ql/autoinfo2.

If a new file system is successfully created, you will see the following output:

Mount the S3QL File System

After you create an S3QL file system, the next step is to mount it.

First, create a local mount point and then use the mount. s3ql command to mount the S3QL file system.

1. $ Mkdir ~ /Mnt_s3ql
2. $ Mount. s3ql s3: // [bucket-name] ~ /Mnt_s3ql

Mounting an S3QL file system does not require privileged users. You only need to ensure that you have the write permission on the mount point.

You can use the -- compress parameter to specify a compression algorithm (such as lzma, bzip2, and zlib ). If this parameter is not specified, lzma is used by default. Note: If you specify a custom compression algorithm, it will only apply to the newly created data object and will not affect existing data objects.

1. $ Mount. s3ql -- compress bzip2 s3: // [bucket-name] ~ /Mnt_s3ql

For performance reasons, the S3QL file system maintains a local file cache, which contains recently accessed (partial or all) files. You can use the "-- cachesize" and "-- max-cache-entries" options to customize the File cache size.

If you want other users to access a mounted S3QL file system, use the "-- allow-other" option.

If you want to export the mounted S3QL file system to another machine through NFS, use the "-- nfs" option.

After running mount. s3ql, check whether the S3QL file system has been mounted successfully:

1. $ Df ~ /Mnt_s3ql
2. $ Mount | grep s3ql

Uninstall S3QL File System

To safely uninstall an S3QL File System (which may contain uncommitted data), run the umount. s3ql command. It will wait until all data (including the part in the cache of the local file system) is successfully transmitted to the backend server. Depending on the amount of data to be written, this process may take some time.

1. $ Umount. s3ql ~ /Mnt_s3ql

View S3QL file system statistics and repair S3QL File System

To view the statistics of the S3QL file system, you can use the s3qlstat command to display information such as the total data size, metadata size, duplicate file deletion rate, and compression ratio.

1. $ S3qlstat ~ /Mnt_s3ql

You can use the fsck. s3ql command to check and repair the S3QL file system. Similar to the fsck command, the file system to be checked must be uninstalled first.

1. $ Fsck. s3ql s3: // [bucket-name]

S3QL Use Case: Rsync backup

Let me end this tutorial with a popular use case: local file system backup. To this end, I recommend using the rsync Incremental backup tool, especially because S3QL provides an rsync encapsulation script (/usr/lib/s3ql/pcp. py ). This script allows you to use multiple rsync processes to recursively copy the directory tree to the S3QL target.

1. $/Usr/lib/s3ql/pcp. py-h

This command will use four concurrent rsync connections to back up ~ All contents in/Documents are sent to an S3QL file system.

1. $/Usr/lib/s3ql/pcp. py-a -- quiet -- processes = 4 ~ /Documents ~ /Mnt_s3ql

These files will be first copied to the local file cache, and then gradually synchronized to the backend server in the background.

To learn more about S3QL, such as automatic mounting, snapshots, and immutable trees, I strongly recommend that you read the official user guide. Please tell me what you think about S3QL and your experience in using any other tools.

This article permanently updates the link address: