AWS S3 log files are uploaded to Elk via the server

View a large number of logs generated by S3 through Elk

First of all, clear the idea

Start by synchronizing the logs with the S3cmd command from S3 and then write the log to the file and show it through elk.

First, install the S3cmd command

installation and simple use of s3cmd tools:

Reference documents

Https://www.cnblogs.com/xd502djj/p/3604783.html https://github.com/s3tools/s3cmd/releases

Download the S3cmd installation package from GitHub first

mkdir/home/tools/&& Cd/home/tools/wget HTTPS://GITHUB.COM/S3TOOLS/S3CMD/RELEASES/DOWNLOAD/V2.0.1/ S3cmd-2.0.1.tar.gztar XF S3CMD-2.0.1.TAR.GZMV s3cmd-2.0.1/usr/local/mv/usr/local/s3cmd-2.0.1/usr/local/s3cmdln-s/ Usr/local/s3cmd/s3cmd/usr/bin/s3cmd

use S3cmd–configure to set key after installation is complete

Basically, access key and secure key are configured to generate the following configuration file

[[email protected] s3cmd]# cat /root/.s3cfg[default]access_key =  AKIAI4Q3PTOQ5XXXXXXX    AWS S3 's access key    must be access_token = add_encoding_exts =add_headers =bucket_location = usca_certs_file =cache_file = check_ssl_certificate = truecheck_ssl_hostname = truecloudfront_host =  Cloudfront.amazonaws.comdefault_mime_type = binary/octet-streamdelay_updates = falsedelete _after = falsedelete_after_fetch = falsedelete_removed = falsedry_run =  falseenable_multipart = trueencrypt = falseexpiry_date =expiry_days = Expiry_prefix =follow_symlinks = falseforce = falseget_continue = falsegpg_ command = /usr/bin/gpggpg_decrypt = % (Gpg_command) s -d --verbose -- No-use-agent --batch --yes --passphrase-fd % (PASSPHRASE_FD) s -o % (output_file) s % (input_file) sgpg_encrypt = % (gpg_ command) s -c --verbose --no-use-agent --batch --yes --passphrase-fd % ( PASSPHRASE_FD) s -o % (output_file) s % (input_file) sgpg_passphrase = aviagamesguess_mime _type = truehost_base = s3.amazonaws.comhost_bucket = % (Bucket) s.s3.amazonaws.comhuman_readable_sizes = falseinvalidate_default_index_on_cf =  Falseinvalidate_default_index_root_on_cf = trueinvalidate_on_cf = falsekms_key =limit  = -1limitrate = 0list_md5 = falselog_target_prefix =long_listing =  falsemax_delete = -1mime_type =multipart_chunk_size_mb = 15multipart_max_chunks  = 10000preserve_attrs = trueprogress_meter = trueproxy_host =proxy_port  = 0put_continue = falserecursive = falserecv_chunk = 65536reduced_redundancy = falserequester_pays =  falserestore_days = 1restore_priority = standardsecret_key =  0UONIJRN9QQHANXXXXXXCZXXXXXXXXXXXX  AWS S3 's secret_key   must be send_chunk =  65536server_side_encryption = falsesignature_v2 = falsesignurl_use_https =  falsesimpledb_host = sdb.amazonaws.comskip_existing = falsesocket_timeout =  300stats = falsestop_on_error = falsestorage_class =urlencoding_mode =  Normaluse_http_expect = falseuse_https = falseuse_mime_magic = trueverbosity  = warningwebsite_endpoint = http://% (bucket) s.s3-website-% (location) s.amazonaws.com/ Website_error =website_index = index.html

Second, S3cmd command after the installation is complete script

#!/bin/bash# Enter S3 Sync directory mkdir /server/s3dir/logs/ -p && cd /server/s3dir/logs/# Place the S3 log list in the S3.log file every 5 minutes #while true#do    /usr/bin/s3cmd ls s3://bigbearsdk /logs/ >s3.log     #执行同步命令确认  s3 is the same as server   log     /usr/bin/ s3cmd  sync --skip-existing s3://bigbearsdk/logs/ ./#done # The day's log sequence is appended to a file      grep $ (date +%f)  s3.log  |sort -nk1,2 |awk -f [/]   ' {print  $NF} '  > date.log    sed -i  ' s#\_#\\_#g '   date.log    sed -i  ' s#<#\\\< #g '  date.log    sed  -i  ' s#\ #\\  #g '  date.log    sed -i  ' s#>#\\\> #g '  date.log##[ -f elk.log ] &&#{#    cat  elk.log  >> elk_$ (date +%f). log#    echo > elk.log#    find / home/tools/ -name elk*.log -mtime +7 |xargs rm -f#}# Append the log of each file to the S3 upload log while read linedo   echo  "$line" |sed  ' s# (#\\\ (#g ' |sed   ' s# #\\\) #g ' | sed  ' s#\_#\\_#g ' |sed  ' s#<#\\\< #g ' |sed  ' s#>#\\\> #g ' |sed   ' s#\ #\\  #g '  >while.log   head -1 while.log |xargs  Cat  >> /server/s3dir/s3elk.logdone < date.log

In this case, the contents of the S3 log are all in the S3elk.log this file and then through the Elk monitoring log

Not to be continued ...

Intending to add Dellinger_blue

AWS S3 log files are uploaded to Elk via the server