Hello All, I am trying to parse Amazon S3 access log that has multiple chunks of the file. When i say multiple chunks i mean there are thousand of separate files with each files containing data in it. The data in these files are not JSON. I am having a tough time determining how to parse these file…

Are you getting anything in stdout?

{ "message" => "789235740a39f4da1f30c076c2dc42e7fdaae8302b896ea3386faa6eeb544c2f niraj-hello-na [23/Jun/2016:13:42:04 +0000] xx.xxx.43.30 arn:aws:iam::xxxxxxxx:user/niraj 0BFACD82FD8581AF REST.GET.LOGGING_STATUS - "GET /?logging HTTP/1.1" 200 - 235 - 6 - "-" "Cloudlytics, aws-sdk-java/1.9.16 Linux/…

Why are you on such an old LS version? You should switch to http protocol?

Mark, The reason i have a old protocol is our current ES is on ES 1.5 and i did not wanted to upgrade it as of now so i started using a compatible version of ES. When you say switch to http protocol, does this mean i should specify in protocol => "http" in output section of Logstash config.

LS and ES are not locked like that, you can run 2.3.X LS with 1.5.X ES. And yes, that is correct.

After upgrading to LS 1.5.3 and adding protocol => http , the s3 logs were getting successfully ingested. Somehow the LS 1.5.0 which i had was getting crashed after few successful runs. Thanks a lot Mark for pointing it out.

Discuss the Elastic Stack

Parse Amazon S3 access log with multiple files

Elastic Stack Logstash

niraj_kumar (Niraj Kumar) July 18, 2016, 3:19am 7

I didn't knew that. Well i would try that for sure and let you know.

Topic		Replies	Views
S3 input and Elasticsearch output in Logstash Logstash	3	2215	July 6, 2017
Docker S3 log error Logstash	6	1006	July 6, 2017
Need help for parsing s3 logs to ELK dashboard Logstash	1	889	June 6, 2018
Passs Json input from s3 files to elasticsearch Logstash	4	1827	July 5, 2019
AWS S3 bucket logs to Elastic search service (AWS) through log stash Logstash	7	888	October 24, 2019

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Parse Amazon S3 access log with multiple files

Related topics