# Parse Amazon S3 access log with multiple files

**URL:** <https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700>\
**Category:** Logstash\
**Created:** [July 17, 2016, 8:29pm UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700 "2016-07-17T20:29:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [July 17, 2016, 8:29pm UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/1 "2016-07-17T20:29:13Z")

</div>

Hello All,

I am trying to parse Amazon S3 access log that has multiple chunks of the file. When i say multiple chunks i mean there are thousand of separate files with each files containing data in it. The data in these files are not JSON. I am having a tough time determining how to parse these files with logstash. Below is my logstash config i am using but apparently i do not see any data been logged with elasticsearch.

input {  
file {  
type =\> "s3-access-log"  
path =\> "/opt/s3logs/logs/\*"  
sincedb\_path =\> "/dev/null"  
start\_position =\> "beginning"  
}  
}  
filter {  
if [type] == "s3-access-log" {  
grok {  
match =\> { "message" =\> "%{S3\_ACCESS\_LOG}" }  
}  
date {  
locale =\> "en"  
match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]  
}  
}  
}  
output {  
elasticsearch {  
host =\> ["elk-prod-02-data01"]  
index =\> "niraj-hello-s3-%{+YYYY.MM.dd}"  
}  
stdout { codec =\> rubydebug }  
}

Any help would be great.

Regards  
Niraj

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 18, 2016, 1:39am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/2 "2016-07-18T01:39:40Z")

</div>

Are you getting anything in stdout?

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [July 18, 2016, 2:34am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/3 "2016-07-18T02:34:59Z")

</div>

{  
"message" =\> "789235740a39f4da1f30c076c2dc42e7fdaae8302b896ea3386faa6eeb544c2f niraj-hello-na [23/Jun/2016:13:42:04 +0000] xx.xxx.43.30 arn:aws:iam::xxxxxxxx:user/niraj 0BFACD82FD8581AF REST.GET.LOGGING\_STATUS - "GET /?logging HTTP/1.1" 200 - 235 - 6 - "-" "Cloudlytics, aws-sdk-java/1.9.16 Linux/3.13.0-36-generic OpenJDK\_64-Bit\_Server\_VM/24.65-b04/1.7.0\_65" -",  
"@version" =\> "1",  
"@timestamp" =\> "2016-06-23T13:42:04.000Z",  
"type" =\> "s3-access-log",  
"host" =\> "elk-prod-02-app01",  
"path" =\> "/opt/s3logs/logs/2016-06-23-14-32-08-63692AC56311762B",  
"owner" =\> "789235740a39f4da1f30c076c2dc42e7fdaae8302b896ea3386faa6eeb544c2f",  
"bucket" =\> "niraj-hello-na",  
"timestamp" =\> "23/Jun/2016:13:42:04 +0000",  
"clientip" =\> "xx.xxx.43.30",  
"requester" =\> "arn:aws:iam::xxxxxxxxxxx:user/jeffm",  
"request\_id" =\> "0BFACD82FD8581AF",  
"operation" =\> "REST.GET.LOGGING\_STATUS",  
"key" =\> "-",  
"verb" =\> "GET",  
"request" =\> "/?logging",  
"httpversion" =\> "1.1",  
"response" =\> 200,  
"bytes" =\> 235,  
"request\_time\_ms" =\> 6,  
"referrer" =\> ""-"",  
"agent" =\> ""Cloudlytics, aws-sdk-java/1.9.16 Linux/3.13.0-36-generic OpenJDK\_64-Bit\_Server\_VM/24.65-b04/1.7.0\_65""  
}  
Jul 18, 2016 2:27:38 AM org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$4 handleException  
WARNING: [logstash-elk-prod-02-app01-7804-9784] failed to send ping to [[#zen\_unicast\_1#][elk-prod-02-app01][inet[elk-prod-02-data01/xxx.16.0.166:9200]]]  
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[elk-prod-02-data01/xxx.16.0.166:9200]][internal:discovery/zen/unicast\_gte\_1\_4] request\_id [174] timed out after [3751ms]  
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:745)

This is what i get now when i added a port to the logstash config. But what i dont understand with the above is that even if it is displaying that it is adding something to the elasticsearch but i do not see any index in elasticsearch been created.

ES Version:- 1.5  
LS Version:- 1.5

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 18, 2016, 3:13am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/4 "2016-07-18T03:13:25Z")

</div>

Why are you on such an old LS version?  
You should switch to http protocol?

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [July 18, 2016, 3:17am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/5 "2016-07-18T03:17:10Z")

</div>

Mark,

The reason i have a old protocol is our current ES is on ES 1.5 and i did not wanted to upgrade it as of now so i started using a compatible version of ES.

When you say switch to http protocol, does this mean i should specify in protocol =\> "http" in output section of Logstash config.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 18, 2016, 3:18am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/6 "2016-07-18T03:18:40Z")

</div>

LS and ES are not locked like that, you can run 2.3.X LS with 1.5.X ES.

And yes, that is correct.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [July 18, 2016, 3:19am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/7 "2016-07-18T03:19:28Z")

</div>

I didn't knew that. Well i would try that for sure and let you know.

---

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [July 18, 2016, 5:29pm UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/8 "2016-07-18T17:29:35Z")

</div>

After upgrading to LS 1.5.3 and adding protocol =\> http , the s3 logs were getting successfully ingested. Somehow the LS 1.5.0 which i had was getting crashed after few successful runs.

Thanks a lot Mark for pointing it out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:47am UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700/9 "2017-07-06T04:47:33Z")

</div>