# Logstash gives OOM & CPU Usage too high when used with S3 Input plugin

**URL:** <https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121>\
**Category:** Logstash\
**Created:** [April 29, 2023, 9:26am UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121 "2023-04-29T09:26:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Utpal\_Brahma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/utpal_brahma/32/120401_2.png) [@Utpal\_Brahma](https://discuss.elastic.co/u/Utpal_Brahma)\
**Post date:** [April 29, 2023, 9:26am UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121/1 "2023-04-29T09:26:49Z")

</div>

Logstash gives out of Memory when S3 plugin is used for a bucket which has already existing tones of files.

 ![Screenshot 2023-04-29 at 2.55.58 PM](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b5fbb84c9d07159cd92dd7907ec748658e77608e.jpeg)  
 ![Screenshot 2023-04-29 at 2.56.10 PM](https://us1.discourse-cdn.com/elastic/original/3X/c/1/c1adacd5e80fb05322d8812e8c557620c4bc5f17.jpeg)  
 ![Screenshot 2023-04-29 at 2.56.21 PM](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a0b39b8442e18ff598b0c6004f313ebecc54b502.jpeg)  
 ![Screenshot 2023-04-29 at 2.57.29 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d841c78cdd761ab7fefd8440a9daee9b95315d14.jpeg)

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 29, 2023, 11:14am UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121/2 "2023-04-29T11:14:01Z")

</div>

> [@Utpal\_Brahma](#):
>
> Logstash gives out of Memory when S3 plugin is used for a bucket which has already existing tones of files.

How much memory you configured for the Logstash heap in the `jvm.options` file? And what does your configuration pipeline looks like?

The s3 input has a lot of issues when dealing with buckets with a lot of files, but I would expected it to be slow, not thrown an OOM

---

<div class="post-metadata">

**Author:** ![Utpal\_Brahma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/utpal_brahma/32/120401_2.png) [@Utpal\_Brahma](https://discuss.elastic.co/u/Utpal_Brahma)\
**Post date:** [April 29, 2023, 4:47pm UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121/3 "2023-04-29T16:47:11Z")

</div>

I am using Heap Size of 5GB, in an ECS container of 1.7vCPU & 8GB memory.

Pipeline Config:

```auto
input {
  s3 {
    region => "${REGION}"
    bucket => "${S3_BUCKET}"
    interval => "120"
    type => "ecs"
   # delete => true
    codec => "json"
    prefix => "2023/"
    gzip_pattern => ".*?$"
    sincedb_path => "/usr/share/logstash/data/plugins/inputs/s3/since_db_file"
  }
}
filter{
    mutate {
      gsub => ["[event][original]", "}{", "},{"]
      gsub => ["[event][original]", "^{", "[{"]
      gsub => ["[event][original]", "}$", "}]"]
    }
    json {
      source => "[event][original]"
      target => "json_message"
    }
    split {
      field => "json_message"
    }
    split{
      field => "[json_message][logEvents]"
    }
    mutate{
      add_field => {
        "log-group" => "%{[json_message][logGroup]}"
        "log-stream" => "%{[json_message][logStream]}"
        "raw-application-log" => "%{[json_message][logEvents][message]}"
      }
    }
    if [raw-application-log] =~ "^{.*}$" {
      json{
        source => "raw-application-log"
        target => "json-application-log"
      }
      if [json-application-log][child_task_id]{
          mutate{
            add_field => {
              "child-task-id" => "%{[json-application-log][child_task_id]}"
            }
          }
      }
      if [json-application-log][child_task_name]{
          mutate{
            add_field => {
              "child-task-name" => "%{[json-application-log][child_task_name]}"
            }
          }
      }
# if [json-application-log][event]{
# mutate{
# add_field => {
# "event" => "%{[json-application-log][event]}"
# }
# }
# }
      if [json-application-log][level]{
          mutate{
            add_field => {
              "log-level" => "%{[json-application-log][level]}"
            }
          }
      }
      if [json-application-log][logger]{
          mutate{
            add_field => {
              "logger" => "%{[json-application-log][logger]}"
            }
          }
      }
      if [json-application-log][parent_task_id]{
          mutate{
            add_field => {
              "parent-task-id" => "%{[json-application-log][parent_task_id]}"
            }
          }
      }
      if [json-application-log][request_id]{
          mutate{
            add_field => {
              "request-id" => "%{[json-application-log][request_id]}"
            }
          }
      }
      if [json-application-log][task_id]{
          mutate{
            add_field => {
              "task-id" => "%{[json-application-log][task_id]}"
            }
          }
      }
      if [json-application-log][tenant]{
          mutate{
            add_field => {
              "tenant" => "%{[json-application-log][tenant]}"
            }
          }
      }
      if [json-application-log][tenant_db_alias]{
          mutate{
            add_field => {
              "tenant-db-alias" => "%{[json-application-log][tenant_db_alias]}"
            }
          }
      }
      if [json-application-log][user_id]{
          mutate{
            add_field => {
              "user-id" => "%{[json-application-log][user_id]}"
            }
          }
      }
      if [json-application-log][timestamp]{
          mutate{
            add_field => {
              "log-timestamp" => "%{[json-application-log][timestamp]}"
            }
          }
      }
    }
    mutate{
      remove_field => [
        "[event][original]",
        "[json_message]",
        "[logEvents]",
        "[messageType]",
        "[owner]",
        "[logGroup]",
        "[logStream]",
        "[subscriptionFilters]",
        "[json-application-log]",
        "[type]"
      ]
    }
    grok {
      match => [
        "log-group",
        "/ecs/%{GREEDYDATA:environment}/%{GREEDYDATA:service}"
      ]
    }
# sleep {
# time => "1" # Sleep 1 second
# every => 300 # on every 100th event
# }
}
output {
  elasticsearch {
    hosts => ["http://${ELASTICSEARCH_HOST}:80"]
    index => "%{environment}"
  }
  stdout { codec => json }
}

```

---

<div class="post-metadata">

**Author:** ![Utpal\_Brahma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/utpal_brahma/32/120401_2.png) [@Utpal\_Brahma](https://discuss.elastic.co/u/Utpal_Brahma)\
**Post date:** [April 29, 2023, 5:25pm UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121/4 "2023-04-29T17:25:38Z")

</div>

Here, is the Logstash metrics. Can any one explain why events received is much less than events emitted.

 ![Screenshot 2023-04-29 at 10.50.56 PM](https://us1.discourse-cdn.com/elastic/original/3X/6/9/69c54fd41b79f1ecc4b9cab3d8891530f9d796c6.jpeg)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 27, 2023, 5:25pm UTC](https://discuss.elastic.co/t/logstash-gives-oom-cpu-usage-too-high-when-used-with-s3-input-plugin/331121/5 "2023-05-27T17:25:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.