Drill integrated open-source S3 storage minio

Drill supports S3 data query, and the new version can be integrated with minio through simple configuration.

Test Run drill with docker refer to https://www.cnblogs.com/rongfengliang/p/9846899.html,minio installation you can also use docker
For minio bucket creation and file upload, you can use the ui or MC

Configuration

• S3 configuration instructions

{  "type": "file",  "connection": "s3a://app/",  "config": {    "fs.s3a.access.key": "<key>",    "fs.s3a.secret.key": "<secret>",    "fs.s3a.endpoint": "<minio server>",    "fs.s3a.connection.ssl.enabled": "false", // default isn‘t  ssl    "fs.s3a.path.style.access": "true"   },  "workspaces": {    "root": {      "location": "/",      "writable": false,      "defaultInputFormat": null,      "allowAccessOutsideWorkspace": false    },    "tmp": {      "location": "/tmp",      "writable": true,      "defaultInputFormat": null,      "allowAccessOutsideWorkspace": false    }  },  "formats": {    "psv": {      "type": "text",      "extensions": [        "tbl"      ],      "delimiter": "|"    },    "csv": {      "type": "text",      "extensions": [        "csv"      ],      "delimiter": ","    },    "tsv": {      "type": "text",      "extensions": [        "tsv"      ],      "delimiter": "\t"    },    "parquet": {      "type": "parquet"    },    "json": {      "type": "json",      "extensions": [        "json"      ]    },    "avro": {      "type": "avro"    },    "sequencefile": {      "type": "sequencefile",      "extensions": [        "seq"      ]    },    "csvh": {      "type": "text",      "extensions": [        "csvh"      ],      "extractHeader": true,      "delimiter": ","    }  },  "enabled": true}

• Query

select * from s3.`root`.`user.json`

• Add data to minio

{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}{"name":"dalong","age":33}

Effect

References

Https://blog.minio.io/query-minio-datastore-with-apache-drill-dcaf71d0cee5? _ Branch_match_id = 520116032687566918
Https://www.cnblogs.com/rongfengliang/p/9846899.html
Http://drill.apache.org/docs/s3-storage-plugin/

Drill integrated open-source S3 storage minio