Logstash cannot read redis data, and logstash can read redis data.

Logstash cannot read redis data, and logstash can read redis data.

A problem occurred when constructing logsatsh + redis + elasticsearch today. After nearly one hour of troubleshooting, the problem was finally solved. Record it.

The environment is like this. A client sends data to redis on the server, and logstash on the server reads redis data and stores it in elasticsearch.

The initial problem is that on the server side, the log sent from the client is not received.

In this way, there may be two problems: first, the logstash of the client fails to send data to redis, and second, the logstash of the server cannot read data from redis.

Troubleshooting 1

 

To avoid data impact judgment in redis, firstClear redisData in:

Go to the redis directory.

Src/redis-cli, connect to redis, execute fulshdb, will delete the currently selected database. Here, the database is actually represented by a KEY and executed:

KEYS *

You can view all the keys. Here the KEY corresponds to the key in our configuration file. Select it and delete it:

select $keyflushdb

In the configuration file, add stdout {} to print the log to the screen.

Because the configured input is file, there is a file in the user's home directory that records the location where the content is read:. sincedb *. To prevent false positives due to the absence of new logs, delete the file and restart Logstash.

In this case, logs are output on the screen. Check whether there is a corresponding key in redis:

KEYS *

If yes, it indicates that the data has been sent to redis. I have it here, so I have ruled out the possibility of the first problem.

Troubleshooting problem 2

To avoid the original data impact of elasticsearchClear elsaticsearchData in:

curl -XDELETE 'http://localhost:9200/_all'

Then, add stdout {} output to the configuration file.

Start logstash and find no log output on the screen .. Now the problem is fixed ..

Solution

Why can't elasticsearch retrieve data from redis? Is it because it is not connected to redis? Or is the key and data_type not correct?

Because redis and the Logstash on the server are on the same machine, the host previously written is the local ip address, and the 127.0.0.1 file written in the official document is changed to 127.0.0.1.

To prevent the key and data_type on the client and server from being matched, check the official documents and read the double quotation marks. Change both the server side to double quotation marks, and copy the key and data_type to the client configuration file.

Restart, OK! By the way, paste the configuration file.

Server:

Client: