Docker logstash keeps heapdumping

Elastic Stack Logstash
1

Hey,

I am using logstash to load a Elasticsearch full of data from text files that i wish to keep the files afterwards as well.

Current Logstash Config.

input {
	beats {
		port => 5044
	}

	tcp {
		port => 5001
	}
	
	file {
                path => "/tmp/data/*/*/server-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/server.db"
                }
                
    file {
                path => "/tmp/data/*/*/headers-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/headers.db"
                }
                
    file {
                path => "/tmp/data/*/*/body-*.txt"
                mode => "read"
                max_open_files => 63979
                file_chunk_size => 129932768
        		sincedb_path => "/tmp/logstash/body.db"
                }
    file {
                path => "/tmp/data/*/*/status-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/status.db"
                }   
                
    file {
                path => "/tmp/data/*/*/favicon-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/favicon.db"
                } 
                
    file {
                path => "/tmp/data/*/*/apps-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/apps.db"
                }   
    file {
                path => "/tmp/data/*/*/bodyhash-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/bodyhash.db"
                }   
          
    file {
                path => "/tmp/data/*/*/bodyhash256-*.txt"
                mode => "read"
                max_open_files => 863979
                file_chunk_size => 9932768
        		sincedb_path => "/tmp/logstash/bodyhash256.db"
                }   
                
                
}

## Add your filters / logstash plugins configuration here

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		user => "elastic"
		password => "changeme"
		ecs_compatibility => disabled
		index => "binaryedge"
	}
	stdout {
    codec => rubydebug
  }
}

Docker-compose

elasticsearch:
    container_name: elasticsearch
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: 7.16.3
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx2g -Xms1g"
      ELASTIC_PASSWORD: changeme
      # Use single node discovery in order to disable production mode and avoid bootstrap checks.
      # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
      
    ulimits:
      memlock:
        soft: -1
        hard: -1
    links:
      - rabbit
      - redis
      - worker
      - db
      - logstash
      
 
  logstash:
    container_name: logstash
    user: root
    build:
      context: logstash/
      args:
        ELK_VERSION: 7.16.3
    volumes:
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,z
      - ./data/:/tmp/data/:ro
      - logstash:/tmp/logstash/:rw
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx12g -Xms8g"

I have about 8 cored and 16gb ram on the system.

Any ideas why logstash keeps crashing

2

Welcome to our community! :smiley:

FYI you are better off keeping Xmx and Xms the same values.
Also please share your Logstash logs :slight_smile:

3

After About 30 mins it will crash and just java heap.

Using bundled JDK: /usr/share/logstash/jdk,
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.,
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties,
[2022-01-31T07:09:11,068][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties,
[2022-01-31T07:09:11,075][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"},
[2022-01-31T07:09:11,097][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"},
[2022-01-31T07:09:11,105][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"},
[2022-01-31T07:09:11,370][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"eabb7c7b-ce3b-455b-91f8-2b677c547f7c", :path=>"/usr/share/logstash/data/uuid"},
[2022-01-31T07:09:12,125][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release.,
Please configure Metricbeat to monitor Logstash. Documentation can be found at: ,
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html,
[2022-01-31T07:09:12,728][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}},
[2022-01-31T07:09:12,874][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.10] failed: Connection refused (Connection refused)"},
[2022-01-31T07:09:12,908][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.10] failed: Connection refused (Connection refused) {:url=>http://elastic:xxxxxx@elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.10] failed: Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"},
[2022-01-31T07:09:12,913][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.10] failed: Connection refused (Connection refused)"},
[2022-01-31T07:09:12,939][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.,
[2022-01-31T07:09:13,058][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false},
[2022-01-31T07:09:13,756][INFO ][org.reflections.Reflections] Reflections took 64 ms to scan 1 urls, producing 119 keys and 417 values ,
[2022-01-31T07:09:15,063][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]},
[2022-01-31T07:09:15,090][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}},
[2022-01-31T07:09:15,108][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.18.0.10] failed: Connection refused (Connection refused)"},
[2022-01-31T07:09:15,198][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x340f8f22 run>"},
[2022-01-31T07:09:16,140][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.94},
[2022-01-31T07:09:16,346][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"},
[2022-01-31T07:09:16,370][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"},
[2022-01-31T07:09:16,397][INFO ][logstash.inputs.tcp      ][main][0c5bf1ca59147f3e9cc92fdd6c9fe9f7f3d1744a14f23765a31916f64f625231] Starting tcp input listener {:address=>"0.0.0.0:5001", :ssl_enable=>false},
[2022-01-31T07:09:16,416][INFO ][filewatch.observingread  ][main][d44df1305c3aa70c717748c47f92f8117809bb8776553c0ffd367fc1a8933b87] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,416][INFO ][filewatch.observingread  ][main][381e455e024f3e1221dd04968e6718ecf347f652d24098f9ea15d6095481c502] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,416][INFO ][filewatch.observingread  ][main][511e777a9013e4c040ecb549dfe7d094971a9f58f20bcf4da7503545cea93382] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,416][INFO ][filewatch.observingread  ][main][aeb5f0c66fe84716b51ca28e80306e0741225ea09254733fc012df459f62a069] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,439][INFO ][filewatch.observingread  ][main][07d1c3c1b3bfd6a594bf10f321498c2ef006d6771d23d1d5128db982b16a86a7] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,442][INFO ][filewatch.observingread  ][main][e380ed38c4cf318da14b74b66f502af3f04e352073808f083b35d7dd4afe8024] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,445][INFO ][filewatch.observingread  ][main][38ecdab2a2cf937dbe47d946763f2a4046f792d1405f0c60d0fc828d94eccfac] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,459][INFO ][filewatch.observingread  ][main][80431e04fbe4ca8c8d1eefc53a7dc84246cef9f60c5f7849a9152b4cdd3b44f8] START, creating Discoverer, Watch with file and sincedb collections,
[2022-01-31T07:09:16,600][INFO ][org.logstash.beats.Server][main][eb30b48ba81bcb13109967448bd23c7db19bd141a80bc35ef42c5dadc986f262] Starting server on port: 5044,
[2022-01-31T07:09:16,739][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]},
[2022-01-31T07:09:20,486][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"},
[2022-01-31T07:09:20,683][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.16.3) {:es_version=>7},
[2022-01-31T07:09:20,686][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7},
[2022-01-31T07:09:21,245][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
4

xms and xmx staying the same has improved now just seeing.

  
[2022-01-31T07:10:13,758][INFO ][filewatch.readmode.handlers.readfile][main][511e777a9013e4c040ecb549dfe7d094971a9f58f20bcf4da7503545cea93382] buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\n", "read_position"=>0, "bytes_read_count"=>3, "last_known_file_size"=>3, "file_path"=>"/tmp/data/google.com/ebs2-uat-app.corp.google.com/server-443.txt"},
[2022-01-31T07:11:22,368][INFO ][filewatch.readmode.handlers.readfile][main][38ecdab2a2cf937dbe47d946763f2a4046f792d1405f0c60d0fc828d94eccfac] buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\n", "read_position"=>0, "bytes_read_count"=>111653, "last_known_file_size"=>111653, "file_path"=>"/tmp/data/shutterfly.com/mathiassen24geislermccrackenhouston165.shutterfly.com/body-443.txt"},
[2022-01-31T07:11:25,598][INFO ][filewatch.readmode.handlers.readfile][main][80431e04fbe4ca8c8d1eefc53a7dc84246cef9f60c5f7849a9152b4cdd3b44f8] buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\n", "read_position"=>0, "bytes_read_count"=>343, "last_known_file_size"=>343, "file_path"=>"/tmp/data/google.com/www2.drive.google.com/apps-80.txt"}

but im sure improving the size of file_chunk_size will fix that.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.