# Filebeat consumes all memory **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [March 13, 2023, 7:42pm UTC](https://discuss.elastic.co/t/filebeat-consumes-all-memory/327592 "2023-03-13T19:42:40Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Radoslav\_Stefanov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radoslav_stefanov/32/118398_2.png) [@Radoslav\_Stefanov](https://discuss.elastic.co/u/Radoslav_Stefanov)\ **Post date:** [March 13, 2023, 7:42pm UTC](https://discuss.elastic.co/t/filebeat-consumes-all-memory/327592/1 "2023-03-13T19:42:40Z")
Hi! I have the following setup to ingest logs from s3. 1. filebeat pulls data from s3 and sends it to logstash. 2. logstash ingests into elastic. The problem is after a while (usually a day or two) filebeat consumes all server memory until oom gets triggered and the kernel kills filebeat. Box has 128gb memory with around 64gb dedicated to filebeat... Bellow is filebeat and logstash configuration. Obscuring keys, endpoints and domains. Please advise how to approach this. Thanks! filebeat ```auto monitoring: enabled: true cluster_uuid: "_7KKV20eSemBTs_BxXfs6A" elasticsearch: hosts: "elasticsearch:9200" username: elastic password: changeme filebeat.queue.mem: events: 4096 flush.min_events: 512 flush.timeout: 30s filebeat.queue.disk: max_size: 100GB filebeat.inputs: - type: aws-s3 bucket_arn: arn number_of_workers: 5 endpoint: https://endpoint access_key_id: key secret_access_key: key tags: ["stage"] - type: aws-s3 bucket_arn: arn ignore_older: 3h clean_inactive: 4h close_inactive: 1m scan_frequency: 2m harvester_limit: 5 number_of_workers: 8 endpoint: https://endpoint.com access_key_id: key secret_access_key: key bucket_list_prefix: "other-domain.com/" file_selectors: - regex: "other-domain.com/202302" - regex: "other-domain.com/202303" - regex: "other-domain.com/202304" - regex: "other-domain.com/202305" - regex: "other-domain.com/202306" - regex: "other-domain.com/202307" - regex: "other-domain.com/202308" - regex: "other-domain.com/202309" - regex: "other-domain.com/202310" - regex: "other-domain.com/202311" - regex: "other-domain.com/202312" - regex: "other-domain.com/2024" tags: ["other"] - type: aws-s3 bucket_arn: arn ignore_older: 3h clean_inactive: 4h close_inactive: 1m scan_frequency: 2m harvester_limit: 5 number_of_workers: 8 endpoint: https://endpoint.com access_key_id: key secret_access_key: key bucket_list_prefix: "domain.com/" file_selectors: - regex: "domain.com/202302" - regex: "domain.com/202303" - regex: "domain.com/202304" - regex: "domain.com/202305" - regex: "domain.com/202306" - regex: "domain.com/202307" - regex: "domain.com/202308" - regex: "domain.com/202309" - regex: "domain.com/202310" - regex: "domain.com/202311" - regex: "domain.com/202312" tags: ["domain"] - type: aws-s3 bucket_arn: arn ignore_older: 3h clean_inactive: 4h close_inactive: 1m scan_frequency: 2m harvester_limit: 5 number_of_workers: 8 endpoint: https://endpoint.com access_key_id: key secret_access_key: key bucket_list_prefix: "more-domain.com/" file_selectors: - regex: "more-domain.com/202302" - regex: "more-domain.com/202303" - regex: "more-domain.com/202304" - regex: "more-domain.com/202305" - regex: "more-domain.com/202306" - regex: "more-domain.com/202307" - regex: "more-domain.com/202308" - regex: "more-domain.com/202309" - regex: "more-domain.com/202310" - regex: "more-domain.com/202311" - regex: "more-domain.com/202312" tags: ["more"] processors: - decode_json_fields: fields: ["message"] process_array: false max_depth: 10 target: "" overwrite_keys: true add_error_key: false - fingerprint: fields: ["message"] target_field: "@metadata._id" setup.kibana.host: "http://kibana:5601" output.logstash: hosts: ["logstash:5044"] ``` logstash ```auto input { beats { port => 5044 include_codec_tag => false client_inactivity_timeout => 120 } } ## Add your filters / logstash plugins configuration here filter { date { match => ["EdgeStartTimestamp", "ISO8601", "UNIX", "UNIX_MS"] } useragent { # ecs_compatibility => v8 source => "ClientRequestUserAgent" target => "ParsedUserAgent" } grok { match => {"ClientRequestPath" => "/%{WORD:api}/%{WORD:api_version}/%{WORD:functionality}/%{WORD:sub_functionality}"} } mutate { convert => {"EdgeResponseCompressionRatio"=>"float" } } if "_grokparsefailure" in [tags] { mutate { remove_tag => ["_grokparsefailure"] add_field => {"_grokparsefailure" => "true"} } } } output { if "other" in [tags] { elasticsearch { hosts => "elasticsearch:9200" user => "elastic" index => "domain-logs-%{+yyyy-MM}" password => "changeme" ecs_compatibility => disabled } } else if "other" in [tags] { elasticsearch { hosts => "elasticsearch:9200" user => "elastic" index => "other-logs-%{+yyyy-MM}" password => "changeme" ecs_compatibility => disabled } } else if "another" in [tags] { elasticsearch { hosts => "elasticsearch:9200" user => "elastic" index => "another-logs-%{+yyyy-MM}" password => "changeme" ecs_compatibility => disabled } } else if "stage" in [tags] { elasticsearch { hosts => "elasticsearch:9200" user => "elastic" index => "stage-logs" password => "changeme" ecs_compatibility => disabled } } } ``` docker stats ```auto CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 3764f82c7c4c filebeat 147.32% 53.78GiB / 125.7GiB 42.78% 49.8GB / 6.74GB 34.6MB / 833GB 40 1aeb8e2c27da nginx 0.00% 29.63MiB / 125.7GiB 0.02% 29.3MB / 35.5MB 51.7MB / 20.5kB 33 671370e801fb zotaelk_logstash_1 0.46% 24.21GiB / 125.7GiB 19.26% 71.9GB / 598GB 3.55TB / 112MB 218 d1cc2eafd36e zotaelk_kibana_1 1.79% 403.2MiB / 125.7GiB 0.31% 1.45GB / 1.7GB 660GB / 4.1kB 12 296ea56bd26f vouch-proxy 0.00% 16.98MiB / 125.7GiB 0.01% 7.5MB / 1.08MB 186GB / 0B 27 f87849d812f6 cloudflared 0.14% 32.68MiB / 125.7GiB 0.03% 93.6MB / 105MB 1.65TB / 0B 37 80d4d5e9955d zotaelk_elasticsearch_1 3.09% 35.19GiB / 125.7GiB 27.99% 599GB / 3.07GB 4.16TB / 1.62TB 271 ``` ![Screenshot 2023-03-13 at 21.40.35](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f6674e44daaf6644eb6a61febe0d3e3419e64035.jpeg) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 19, 2023, 10:39pm UTC](https://discuss.elastic.co/t/filebeat-consumes-all-memory/327592/2 "2023-03-19T22:39:45Z")
I don't know a lot about this area, but how many s3 buckets are you pulling in from? Cause Filebeat needs to track these and that might be what is taking all the memory? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 17, 2023, 12:40am UTC](https://discuss.elastic.co/t/filebeat-consumes-all-memory/327592/3 "2023-04-17T00:40:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.