Logstash 5.4.0 stuck/hang after processing certain amount of data (http-output plugin)

Elastic Stack Logstash
1

Hi All,

I am using Logstash 5.4.0 for logs parsing and data ingestion. Issue is that the Logstash becomes unresponsive after processing around 2M log lines.

Hardware Configuration is:
• RHEL 7
• 16 Cores CPU, 64 GB RAM
• Java 8

Logstash Config:
• Logstash version 5.4.0
• -Xmx=4g
• -Xms=4g
• pipeline.batch.size: 125
• pipeline.output.workers: 16
• pipeline.batch.delay: 5

logstash conf code for output section:

http {
http_method => "post"
url => "http://localhost:8080/test"
format => "message"
headers => {"Content-Type" => "application/json"}
content_type => "application/json"
message => '%{message}'
pool_max => 400
pool_max_per_route => 200
}

Any help on this will be highly appreciated.

Also attaching the logstash stats in text file for the reference.

JVM Stats:
{
"host": "test.dev.fco",
"version": "5.4.0",
"http_address": "127.0.0.1:9600",
"id": "9dbbab77-5708-4fb5-9164-8186218c43ea",
"name": "test.dev.fco",
"jvm": {
"threads": {
"count": 88,
"peak_count": 1305
},
"mem": {
"heap_used_in_bytes": 394557488,
"heap_used_percent": 9,
"heap_committed_in_bytes": 758067200,
"heap_max_in_bytes": 4181590016,
"non_heap_used_in_bytes": 135493912,
"non_heap_committed_in_bytes": 144883712,
"pools": {
"survivor": {
"peak_used_in_bytes": 17891328,
"used_in_bytes": 9084368,
"peak_max_in_bytes": 113377280,
"max_in_bytes": 113377280,
"committed_in_bytes": 17891328
},
"old": {
"peak_used_in_bytes": 489065328,
"used_in_bytes": 323718520,
"peak_max_in_bytes": 3160866816,
"max_in_bytes": 3160866816,
"committed_in_bytes": 597045248
},
"young": {
"peak_used_in_bytes": 143130624,
"used_in_bytes": 61754600,
"peak_max_in_bytes": 907345920,
"max_in_bytes": 907345920,
"committed_in_bytes": 143130624
}
}
},
"gc": {
"collectors": {
"old": {
"collection_time_in_millis": 14120,
"collection_count": 229
},
"young": {
"collection_time_in_millis": 151233,
"collection_count": 11051
}
}
},
"uptime_in_millis": 2579583
}
}

Hot Threads:
::: {}
Hot threads at 2017-07-07T01:42:56-06:00, busiestThreads=3:

46.61 % of cpu usage, state: waiting, thread name: '[main]>worker4'
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:460)
org.jruby.RubyThread$SleepTask.run(RubyThread.java:1050)
org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066)
org.jruby.RubyThread.wait_timeout(RubyThread.java:1414)
org.jruby.ext.thread.Queue.pop(Queue.java:152)
org.jruby.ext.thread.Queue.pop(Queue.java:127)
org.jruby.ext.thread.Queue$INVOKER$i$pop.call(Queue$INVOKER$i$pop.gen)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
rubyjit.LogStash::Outputs::Http$$send_events_53b66f5675683db7f49396d05b16a7190f3381571560911714.chained_1_rescue_2$RUBY$SYNTHETIC__file__(/opt/ca/axa/logparser/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-http-4.1.0/lib/logstash/outputs/http.rb:141)

36.53 % of cpu usage, state: waiting, thread name: '[main]>worker5'
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:460)
org.jruby.RubyThread$SleepTask.run(RubyThread.java:1050)
org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066)
org.jruby.RubyThread.wait_timeout(RubyThread.java:1414)
org.jruby.ext.thread.Queue.pop(Queue.java:152)
org.jruby.ext.thread.Queue.pop(Queue.java:127)
org.jruby.ext.thread.Queue$INVOKER$i$pop.call(Queue$INVOKER$i$pop.gen)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
rubyjit.LogStash::Outputs::Http$$send_events_53b66f5675683db7f49396d05b16a7190f3381571560911714.chained_1_rescue_2$RUBY$SYNTHETIC__file__(/opt/ca/axa/logparser/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-http-4.1.0/lib/logstash/outputs/http.rb:141)

6.56 % of cpu usage, state: waiting, thread name: '[main]>worker2'
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:460)
org.jruby.RubyThread$SleepTask.run(RubyThread.java:1050)
org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066)
org.jruby.RubyThread.wait_timeout(RubyThread.java:1414)
org.jruby.ext.thread.Queue.pop(Queue.java:152)
org.jruby.ext.thread.Queue.pop(Queue.java:127)
org.jruby.ext.thread.Queue$INVOKER$i$pop.call(Queue$INVOKER$i$pop.gen)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
rubyjit.LogStash::Outputs::Http$$send_events_53b66f5675683db7f49396d05b16a7190f3381571560911714.chained_1_rescue_2$RUBY$SYNTHETIC__file__(/opt/ca/axa/logparser/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-output-http-4.1.0/lib/logstash/outputs/http.rb:141)

Thanks in advance.

Ashutosh

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.