Logstash works well with an error

Chunbo_Liao · August 24, 2018, 6:06am

I used the logstash with the version of 5.6.3, after a few time later. there is an error message in logs/logstash-plain.log. the error is :[ERROR][logstash.filters.ruby ] Ruby exception occurred: Regexp Interrupted. but my logstash is still work. does this a matter?

the whole logs is:
[2018-08-24T13:19:46,193][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/Miracle/elk/logstash-5.6.3/modules/fb_apache/configuration"}
[2018-08-24T13:19:46,198][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/Miracle/elk/logstash-5.6.3/modules/netflow/configuration"}
[2018-08-24T13:19:51,806][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.2.10:9200/]}}
[2018-08-24T13:19:51,808][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.2.10:9200/, :path=>"/"}
[2018-08-24T13:19:51,919][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.56.233.182:9200/"}
[2018-08-24T13:19:52,001][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-08-24T13:19:52,006][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-08-24T13:19:52,020][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.2.10:9200"]}
[2018-08-24T13:19:52,440][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2000}
[2018-08-24T13:19:52,664][INFO ][logstash.pipeline ] Pipeline main started
[2018-08-24T13:19:52,811][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-24T13:45:15,014][ERROR][logstash.filters.ruby ] Ruby exception occurred: Regexp Interrupted
[2018-08-24T14:09:31,460][ERROR][logstash.filters.ruby ] Ruby exception occurred: Regexp Interrupted

due to no other error messages, how can I debug this error?

magnusbaeck · August 24, 2018, 6:18am

It looks like you have a ruby filter that's taking too long to filter events. Without knowing anything about your configuration that's really all I can say.

Chunbo_Liao · August 24, 2018, 6:38am

you mean that there are some low efficiency ruby code in my conf files?

there are 5 ruby code in my conf files .
1:
ruby{
code =>"
if !(event.get('MessageType').nil?)
event.set('MessageType',event.get('MessageType').gsub(' ',''))
end
if !(event.get('MessageName').nil?)
event.set('MessageName',event.get('MessageName').gsub(' ',''))
end
if !((event.get('message').split('ticket:')[1]).nil?)
prioString = event.get('message').split('ticket:')[1]
prioArray = prioString.split(',')
if !(prioArray.nil?)
for prio in prioArray do
prioName='undefined'
if !((prio.split(':')[0]).nil?)
prioName=prio.split(':')[0].gsub(' ','')
end
if !((prio.split(':')[1]).nil?)
prioLateCount=prio.split(':')[1].to_i
event.set(prioName, prioLateCount)
end
end
end
end"
}
2:
ruby{
code =>"
if !(event.get('MessageType').nil?)
event.set('MessageType',event.get('MessageType').gsub(' ',''))
end
if !((event.get('message').split('-')[-1]).nil?)
gcString = event.get('message').split('-')[-1].gsub(' ','')
gcArray = gcString.split(',')
if !(gcArray.nil?)
for gcItem in gcArray
if !((gcItem .split(':')[1]).nil?) and !((gcItem .split(':')[0]).nil?)
gcName=gcItem.split(':')[0]
gcCount=gcItem.split(':')[1].to_i
event.set(gcName, gcCount)
end
end
end
end"
}

3:
ruby{
code =>"
if !(event.get('MessageType').nil?)
event.set('MessageType',event.get('MessageType').gsub(' ',''))
end
if !(event.get('MessageName').nil?)
event.set('MessageName',event.get('MessageName').gsub(' ',''))
end
if !((event.get('message').split('SYSTEM INFO - ')[1]).nil?)
tempString = event.get('message').split('SYSTEM INFO - ')[1]
if !((tempString.split(':')[1]).nil?)
loadString=(tempString.split(':')[1]).gsub(' ','')
loadArray = loadString.split(',')
if !(loadArray.nil?)
for load in loadArray do
if !((load.split('-')[0]).nil?) and !((load.split('-')[1]).nil?)
loadName=load.split('-')[0]
loadNameCount=load.split('-')[1].to_f
event.set(loadName, loadNameCount)
end
end
end
end
end"
}

4:
ruby{
code =>"
if !(event.get('MessageType').nil?)
event.set('MessageType',event.get('MessageType').gsub(' ',''))
end
if !(event.get('MessageName').nil?)
event.set('MessageName',event.get('MessageName').gsub(' ',''))
end
if !((event.get('message').split('SYSTEM INFO - ')[1]).nil?)
tempString = event.get('message').split('SYSTEM INFO - ')[1]
if !((tempString.split('-')[1]).nil?)
loadString=(tempString.split('-')[1]).gsub(' ','')
loadArray = loadString.split(',')
if !(loadArray.nil?)
for load in loadArray do
if !((load.split('-')[0]).nil?) and !((load.split('-')[1]).nil?)
loadName=load.split('-')[0].gsub(' ','')
loadNameCount=load.split('-')[1].to_f
event.set(loadName, loadNameCount)
end
end
end
end
end"
}

5:
ruby{
code =>"
if !(event.get('MessageType').nil?)
event.set('MessageType',event.get('MessageType').gsub(' ',''))
end
if !(event.get('MessageName').nil?)
event.set('MessageName',event.get('MessageName').gsub(' ',''))
end
if (event.get('MessageType') != 'DMXForwardedMessages') and !(event.get('message').split('Cause:')[1].nil?)
causeString = event.get('message').split('Cause:')[1]
causeArray = causeString.split(',')
if !(causeArray.nil?)
for i in causeArray do
temp = i.split('-')
causeID='undefined'
if !(temp[0].nil?)
causeID = temp[0]
end
if !(temp[1].nil?)
causeValue = temp[1].to_i
event.set(causeID, causeValue)
end
end
end
elsif (event.get('MessageType') == 'ProcedureStatistics')
procLateArray=event.get('message').split(',')
if !(procLateArray.nil?)
for proc in procLateArray do
if proc.include?'Latency'
procLate='undefined'
if !((proc.split(':')[0]).nil?)
procLate=proc.split(':')[0].gsub(' ','')
end
if !((proc.split(':')[1]).nil?)
procLateCount=proc.split(':')[1].to_i
event.set(procLate, procLateCount)
end
end
end
end
end"
}

magnusbaeck · August 24, 2018, 7:45am

you mean that there are some low efficiency ruby code in my conf files?

That would've been my guess, but your ruby filters seems pretty straight-forward so I don't know what's going on.

Chunbo_Liao · August 24, 2018, 8:41am

now, more errors comeout:
Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"interrupted waiting for mutex: null", "backtrace"=>["org/jruby/ext/thread/Mutex.java:94:in lock'", "org/jruby/ext/thread/Mutex.java:147:insynchronize'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/metric_store.rb:58:in fetch_or_store'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/collector.rb:51:inget'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/collector.rb:36:in push'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/metric.rb:22:inincrement'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/namespaced_metric.rb:24:in increment'", "/home/11thone/elk/logstash-5.6.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.3/lib/logstash/filters/grok.rb:313:infilter'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filters/base.rb:164:inmulti_filter'", "org/jruby/RubyArray.java:1613:in each'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filters/base.rb:161:inmulti_filter'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filter_delegator.rb:46:in multi_filter'", "(eval):12259:ininitialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):12253:ininitialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):12277:ininitialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):12274:ininitialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):1978:infilter_func'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:in filter_batch'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:inworker_loop'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:in start_workers'"]} [2018-08-24T16:41:15,093][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ConcurrencyError: interrupted waiting for mutex: null>, :backtrace=>["org/jruby/ext/thread/Mutex.java:94:inlock'", "org/jruby/ext/thread/Mutex.java:147:in synchronize'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/metric_store.rb:58:infetch_or_store'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/collector.rb:51:in get'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/collector.rb:36:inpush'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/metric.rb:22:in increment'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/instrument/namespaced_metric.rb:24:inincrement'", "/home/11thone/elk/logstash-5.6.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-3.4.3/lib/logstash/filters/grok.rb:313:in filter'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filters/base.rb:145:indo_filter'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/filter_delegator.rb:46:inmulti_filter'", "(eval):12259:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):12253:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):12277:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):12274:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):1978:in filter_func'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:398:infilter_batch'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:379:in worker_loop'", "/home/11thone/elk/logstash-5.6.3/logstash-core/lib/logstash/pipeline.rb:342:instart_workers'"]}

system · September 21, 2018, 8:42am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.