# Filebeat Amazon ECS log rotation issue **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 23, 2019, 4:51pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872 "2019-07-23T16:51:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![justkind](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@justkind](https://discuss.elastic.co/u/justkind)\ **Post date:** [July 23, 2019, 4:51pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872/1 "2019-07-23T16:51:21Z")
Hi Elastic Team, Sorry to bother y'all, but I've been running into an issue using Filebeat on Amazon ECS and would appreciate any help. Summary: `Filebeat 6.8.1` deployed on each individual ECS host instance and forwards logs to an Amazon Elasticache Redis cluster where log events are pulled in by Logstash and the Redis input plugin. The Filebeat container itself is configured with the awslogs driver to send its own logs to Cloudwatch Logs and is configured to forward all docker container logs. I am seeing my Filebeat ECS tasks/containers spiking up to almost max memory for a long period of time before eventually being rotated out by ECS. When the memory spikes, we see the following error and a large amount of blank lines: `ERROR log/harvester.go:282 Read line error: parsing CRI timestamp: parsing time "` ![46%20AM](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5afdc29231a93ac10b4b0c656593f35b76839476.png) ![07%20AM](https://us1.discourse-cdn.com/elastic/original/3X/0/3/03b305dc7070bbcff802c8eae2ad6923774e9a38.png) Looking into the issue, the timestamp of the errors appear to coincide with the log rotation of the Amazon ecs-init agent that runs on each host instance, specifically the gz compression step at `Jul 21 03:16` ``` -rw-r----- 1 root root 9634140 Jul 23 15:34 e905f2afd21d6b423afa80a0101097018ab50783f73be7033cb6a80aa00850f2-json.log -rw-r----- 1 root root 16000151 Jul 23 00:16 e905f2afd21d6b423afa80a0101097018ab50783f73be7033cb6a80aa00850f2-json.log.1 -rw-r----- 1 root root 16000165 Jul 21 23:41 e905f2afd21d6b423afa80a0101097018ab50783f73be7033cb6a80aa00850f2-json.log.2 -rw-r----- 1 root root 185681 Jul 21 03:16 e905f2afd21d6b423afa80a0101097018ab50783f73be7033cb6a80aa00850f2-json.log-20190721.gz -rw-r----- 1 root root 16000252 Jul 21 00:25 e905f2afd21d6b423afa80a0101097018ab50783f73be7033cb6a80aa00850f2-json.log.3 ``` Could anyone take a look at my below configuration and let me know if these errors are due to a misconfiguration on my part? I believe my configuration is not handling log rotation well and am unsure of how to configure it to handle this. Any assistance would be greatly appreciated. Thanks for your time, ``` filebeat.config: modules: path: ${path.config}/modules.d/*.yml reload.enabled: false filebeat.autodiscover: providers: - type: docker templates: config: - type: docker containers.ids: - "${data.docker.container.id}" multiline.pattern: '^[[:space:]]+(at|\.{3})|^Caused by:|^org.springframework|^java.|\\t+(at|\.{3})' multiline.negate: false multiline.match: after filebeat.inputs: - type: docker containers.ids: - "*" processors: - add_docker_metadata: ~ multiline.pattern: '^[[:space:]]+(at|\.{3})|^Caused by:|^org.springframework|^java.|\\t+(at|\.{3})' multiline.negate: false multiline.match: after processors: - add_cloud_metadata: ~ output.redis: hosts: ["redis-filebeat:6379"] key: "filebeat_beta" db: 0 timeout: 5 logging.level: error logging.to_files: false ``` ---
**Author:** ![justkind](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@justkind](https://discuss.elastic.co/u/justkind)\ **Post date:** [July 23, 2019, 8:23pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872/2 "2019-07-23T20:23:32Z")
Another thing I wanted to add is that I believe I saw this issue a few weeks ago when another application did some log rotation **without** compression but am unable to get proof right now as it was a while ago. I assumed it was an issue with the logging level as it was set to INFO at the time resulting in lots of blank log entries so I changed it to ERROR, but unfortunately the error persists. I will add the following line to my docker input to ignore the .gz files to see if that fixes the issue, but I guess the main question in this post is if anyone knows what's happening with the increased memory usage and blank log lines, in this case, specifically because of a log rotated out and compressed ``` exclude_files: ['\.gz$'] ``` ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [July 25, 2019, 5:34pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872/3 "2019-07-25T17:34:37Z")
Hello, I believe this was fixed using the new "container" input type see the relevant issues [https://github.com/elastic/beats/pull/12162](https://github.com/elastic/beats/pull/12162) ---
**Author:** ![justkind](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@justkind](https://discuss.elastic.co/u/justkind)\ **Post date:** [July 25, 2019, 8:51pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872/4 "2019-07-25T20:51:19Z")
Hi @pierhugues , Thanks for the reply. Great to see that it is fixed in the future container input, unfortunately my Elasticsearch and Kibana is stuck at version 6.6 due to a dependency on the Sentinl alerting plugin which as of right now has no plans to upgrade to 7.x. Is there anything I can do to get this working with the current versions and do you think it is related to log rotation or something else? Is upgrading to 7.2+ to get the container input the only fix for this issue? I added the exclude\_files for .gz yesterday and am still waiting to see if the error comes back because I'm still not sure if log rotation is the root cause, but I'll update this ticket if I see anything. Thanks for your time, ---
**Author:** ![justkind](https://avatars.discourse-cdn.com/v4/letter/j/919ad9/32.png) [@justkind](https://discuss.elastic.co/u/justkind)\ **Post date:** [August 6, 2019, 2:59pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872/5 "2019-08-06T14:59:20Z")
Hey @pierhugues, The issue persisted even with the exclude\_files unfortunately, but I found out that our instance had logrotate running at certain times and the log rotation from that seemed to be the root cause. I removed logrotate on our docker logs and have been monitoring for about a week and haven't seen any issues. Just wanted to let you and the team know in case this is something you are interested in looking into, but this issue seems to have been resolved. Thank you ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2019, 2:59pm UTC](https://discuss.elastic.co/t/filebeat-amazon-ecs-log-rotation-issue/191872/6 "2019-09-03T14:59:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.