Memsql FileSystem Pipeline Trial

Some features like drill, such as s3,file ...

Create file Pipeline

• Prepare file

mkdir -p /opt/db/touch books.txt内容如下：The Catcher in the Rye, J.D. Salinger, 1945Pride and Prejudice, Jane Austen, 1813Of Mice and Men, John Steinbeck, 1937Frankenstein, Mary Shelley, 1818

• Create a table

memsqlCREATE DATABASE books;USE books;CREATE TABLE classic_books(title VARCHAR(255),author VARCHAR(255),date VARCHAR(255));

• Create pipeline

CREATE PIPELINE libraryAS LOAD DATA FS ‘/opt/db/*‘INTO TABLE `classic_books`FIELDS TERMINATED BY ‘,‘;

Enable pipeline

• Start

START PIPELINE library;

• View status

SHOW PIPELINES;

Test results

Several questions

• Paused due to error. Run START PIPELINE or consider setting Pipelines_stop_on_error to False

配置参数修改SET GLOBAL pipelines_stop_on_error = false;

• Note the permissions of the file, and the file must be each node, or you will not see the data (I did not pay attention to the master, there is a problem)
• Always have the following tips

Data volume has significantly changed since the last time ANALYZE TABLE was run. Run <a target="_blank" href="https://docs.memsql.com/ops-redir/analyze/?utm_source=ops&amp;utm_medium=link&amp;utm_campaign=ref" data-reactid=".0.3.1.2.0.2.0.0.0.3" style="background:rgb(250, 250, 250);color:rgb(18, 135, 186);text-decoration:none;font-size:1.067rem;"><b data-reactid=".0.3.1.2.0.2.0.0.0.3.0">ANALYZE TABLE</b></a> on each table to improve query performance andrefresh schema.

Workaround

按照提示操作即可，这个可能和我的系统没有进行参数优化有关，具体的可以参考下面的资料的安装最佳实践

Resources

https://docs.memsql.com/memsql-pipelines/v6.0/filesystem-pipelines-quickstart/
https://docs.memsql.com/memsql-pipelines/v6.0/filesystem-pipelines-overview/
https://docs.memsql.com/tutorials/v6.0/installation-best-practices/

Memsql FileSystem Pipeline Trial