Out of memory error with logstash 7.6.2

Elastic Stack Logstash
,
1

Hi everyone,
I have a Logstash 7.6.2 docker that stops running because of memory leak. After each pipeline execution, it looks like Logstash doesn't release memory.
What should I do to identify the source of the problem?
How can I solve it?

Here is the error I see in the logs. These are just the 5 first lines of the Traceback. I uploaded the rest in a file in my github there.

 logstash    | [2020-04-08T18:15:42,960][INFO ][logstash.outputs.file    ][rawweb] Closing file /output/web_data.json
 logstash    | [2020-04-08T18:15:43,353][ERROR][org.logstash.Logstash    ] java.lang.OutOfMemoryError: Java heap space
 logstash    | [2020-04-08T18:15:43,367][ERROR][org.logstash.execution.WorkerLoop][rawclient] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
 logstash    | org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `pop' for nil:NilClass
 logstash    |   at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.awesome_print_minus_1_dot_7_dot_0.lib.awesome_print.inspector.awesome(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/inspector.rb:117) ~[?:?]

Here the docker-compose.yml I used to configure my Logstash Docker

version: '2.4'    
services:
      logstash:
        image: docker.elastic.co/logstash/logstash:7.6.2
        container_name: logstash
        environment:
          LS_JAVA_OPTS: "-Xmx7g -Xms4g"
          REQUEST_FREQUENCY: 600 #seconds
        volumes:
          - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
          - ./logstash/pipelines.yml:/usr/share/logstash/config/pipelines.yml
          - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
          - ./logstash/tests:/testscripts:ro
          - /root/logstash_output/:/output/
        ports:
          - "9600:9600"
        mem_limit: 7000M
        mem_reservation: 100M

My pipelines.yml file

- pipeline.id: rawclient
  path.config: "/usr/share/logstash/pipeline/logclient.conf"
  pipeline.batch.size: 10000000

 - pipeline.id: rawweb
   path.config: "/usr/share/logstash/pipeline/logweb.conf"
   pipeline.batch.size: 10000000

One of my .conf files. Basically, it executes a .sh script containing a curl request. The resulte of this request is the input of the pipeline. Treatments are made. Then results are stored in file. The two pipelines do the same, the only difference is the curl request that is made.

input {
  exec {
    command => "bash /testscripts/logclient_1.sh"
    codec => "json"
    interval => "600"
  }
}

filter {
    mutate {
        rename => ["connection/start_time", "start_time" ]
        rename => ["connection/end_time", "end_time" ]
        rename => ["connection/duration", "duration" ]
        rename => ["connection/destination_ip_address", "destination_ip_address" ]
        rename => ["connection/status", "status" ]
        rename => ["device/last_ip_address", "last_ip_address" ]
        rename => ["user/sid", "sid" ]
        # rename => ["binary/application_category", "application_category" ]
        rename => ["binary/application_name", "application_name" ]
        rename => ["binary/executable_name", "executable_name" ]
        remove_field => ["@timestamp"]
        remove_field => ["@version"]
        add_field => { "connection_type" => "client" }
    }
}
output {
  file {
   path => "/output/client_data.json"
   codec => "json"
 }
  stdout {
   codec => rubydebug
 }
}

My logstash.yml file

http.host: "0.0.0.0"
xpack.monitoring.enabled: false

Thanks for all the help :slightly_smiling_face:

2

I also posted my problem on stack overflow here and I got a solution.

The problem came from the high value of batch size. That was two much data loaded in memory before executing the treatments. It caused heap overwhelming.

So I reduced batch size

That's it ^^

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.