# Docker S3 log error

**URL:** <https://discuss.elastic.co/t/docker-s3-log-error/58634>\
**Category:** Logstash\
**Created:** [August 22, 2016, 11:17pm UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634 "2016-08-22T23:17:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![luisgzvs](https://avatars.discourse-cdn.com/v4/letter/l/85e7bf/32.png) [@luisgzvs](https://discuss.elastic.co/u/luisgzvs)\
**Post date:** [August 22, 2016, 11:17pm UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/1 "2016-08-22T23:17:32Z")

</div>

I'm trying to analice an asset log from AWS S3 bucket, I have a docker container with the ELK stack integrated, my logstash config file is this one:

input {  
file {  
path =\> "/home/\*"  
start\_position =\> beginning  
}  
}

filter {  
grok {  
match =\> { "message" =\> "%{S3\_ACCESS\_LOG}"}  
}

date {  
locale =\> "en"  
match =\> ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]  
}  
}

output {  
elasticsearch { hosts =\> ["localhost"] }  
stdout { codec =\> rubydebug }  
}

I have multiple logs downloaded in my computer, for that reason I use the path "/home/\*", my log info looks like this one:

aac4092fe3b00a5f6f84224f4694f4c46c9f2d147d9d97ddbe7aed04dfd9e99d [assets.quefilo.com](http://assets.quefilo.com) [21/Jul/2016:22:28:16 +0000] 10.219.249.29 arn:aws:iam::038493190737:user/josue.murillo 9D5242A3E8ABC2C4 REST.GET.NOTIFICATION - "GET /?notification HTTP/1.1" 200 - 115 - 23 - "-" "aws-internal/3" -

and I'm using this command to run logstash config file: /opt/logstash/bin/logstash -f /etc/logstash/conf.d/

but logstash is not creating the elasticsearch index, and is not returning any results in terminal, so my question is, why index aren't being created? thank for help!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 23, 2016, 5:33am UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/2 "2016-08-23T05:33:44Z")

</div>

If the files are older than 24 hours you need to adjust the file input's `ignore_older` option. Apart from that I'm pretty sure there are clues in the logs. You may have to increase the logging verbosity by starting Logstash with `--verbose` or even `--debug`.

---

<div class="post-metadata">

**Author:** ![luisgzvs](https://avatars.discourse-cdn.com/v4/letter/l/85e7bf/32.png) [@luisgzvs](https://discuss.elastic.co/u/luisgzvs)\
**Post date:** [August 23, 2016, 4:32pm UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/3 "2016-08-23T16:32:27Z")

</div>

Thanks a lot @magnusbaeck using ignore\_older option now logs are being mapped, I've improved my log configuration file to this one:

input {  
file {  
path =\> "/home/\*"  
start\_position =\> beginning  
ignore\_older =\> 0  
}  
}

filter {  
grok {  
match =\> { "message" =\> "%{SYSLOG5424SD:timestamp}._%{IP} arn:aws:iam%{IP}._:user%{URIPATHPARAM} .\* %{JAVACLASS:method}"}  
}

date {  
match =\> ["timestamp" , "[dd/MMM/YYYY:HH:mm:ss Z]"]  
target =\> "@timestamp"  
}

}

output {  
elasticsearch { hosts =\> ["localhost"] }  
stdout { codec =\> rubydebug }  
}

and I can't get the @timestamp from message, what I'm doing wrong?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 23, 2016, 5:37pm UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/4 "2016-08-23T17:37:01Z")

</div>

As discussed in [S3 assets log](https://discuss.elastic.co/t/s3-assets-log/58401) your grok filter isn't working.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 24, 2016, 6:11am UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/6 "2016-08-24T06:11:59Z")

</div>

What do you have so far that isn't working?

---

<div class="post-metadata">

**Author:** ![luisgzvs](https://avatars.discourse-cdn.com/v4/letter/l/85e7bf/32.png) [@luisgzvs](https://discuss.elastic.co/u/luisgzvs)\
**Post date:** [August 24, 2016, 2:44pm UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/7 "2016-08-24T14:44:09Z")

</div>

I can't get timestamp from log message, but logs are being mapped and can see them in kibana, using configuration mentioned above, and you told me my grok filter wasn't working, but what should be proper grok expression for log mentioned above, to get timestamp from log message?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:41am UTC](https://discuss.elastic.co/t/docker-s3-log-error/58634/8 "2017-07-06T04:41:50Z")

</div>