Logstash Service Restart continuously

Hi,

All of a sudden my logstash service is restarting every 3 minutes and getting the below error

[FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (LoadError) Could not
load FFI Provider: (NotImplementedError) FFI not available: null
See http://jira.codehaus.org/browse/JRUBY-4583
org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
See http://jira.codehaus.org/browse/JRUBY-4583
at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:201) ~[jruby-complete-9.2.20.1.
jar:?]
at RUBY.(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/ffi.rb:11) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share
/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.module:LibC(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
at RUBY.(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share
/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]

The system was up and running since years and by 12 AM on OCT 10th the issues started and above error is observed in the logs.

I am not able to figure out where is the problem

Hello,

Please share more logs, you should have more log lines above or below that one that would help with the troulebshoot.

Also, use the preformatted text option to share the log content, the </> button.

Which version are you using and what changed your system? Did you updated anything?

Hi,

Here are the details..

Logstash version is 7.16.3.

[2023-10-10T07:19:23,340][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T07:19:23,447][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T07:19:23,506][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logst
ash/modules/fb_apache/configuration"}

by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:05:28,239][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2023-10-10T13:07:46,274][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T13:07:46,401][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:07:46,546][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2023-10-10T13:09:49,361][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T13:09:49,400][INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:09:49,496][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2023-10-10T13:12:13,344][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T13:12:13,384][INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:12:13,471][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exittype or paste code here
type or paste code here

Just to add since it was happening during midnight today I don't think someone explicitly changed.. However I am checking more ..

Yeah, nothing very helpful in this log.

Can you stop the logstash service and start it again to get fresh logs from the beginning?

Just systemctl stop logstash and after it stopes, systemctl start logstash, then get the fresh logs from the beginning to see if it brings more helpful information.

Also, what does your logstash.yml file looks like? Please share its content.

Do you have free space in logstash data path and /tmp as well?

Hi,

Thank you for the help.

I tried to stop and start the logstash service which generated new logfile. Unfortunately it is exiting with similar message.

type or paste [2023-10-10T13:55:31,267][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T13:55:31,287][INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:55:31,340][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2023-10-10T13:57:22,039][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T13:57:22,079][INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:57:22,177][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
[2023-10-10T13:59:36,929][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-10T13:59:36,968][INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jru
by 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-10-10T13:59:37,075][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]code here

My logstash.yml looks as below.

type orpath.data: /var/lib/logstash

log.level: DEBUG
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: []

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["https://<endpoint>:9200"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xxxx

pipeline.ecs_compatibility: disabled
xpack.monitoring.elasticsearch.ssl.verification_mode: none paste code here

Yes, I see no specific space issue in the system.

Can't get any idea from the logs of what could be the issue.

Can you check the system logs? It would be /var/log/messages or /var/log/syslog depending on the distribution.

Stop and Start again and check the system logs to see if there is any hint of the issue.

You can also try to start it manually to see if there is any hint of the issue.

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings="/etc/logstash"

Hi,

I tried to find some more logs

type or paste [2023-10-10T15:08:25,361][ERROR][logstash.javapipeline    ] Pipeline error {:pipeline_id=>"main", :exception=>#<Manticore::Unknown
Exception: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty>, :bac
ktrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/response.rb:36:in `block in initia
lize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.1-java/lib/manticore/response.rb:79:in `call'", "/usr/sh
are/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_clie
nt/manticore_adapter.rb:74:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearc
h-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `perform_request_to_url'", "/usr/share/logstash/vendor
/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:235:in `
health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash
/outputs/elasticsearch/http_client/pool.rb:242:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/
logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/p
ool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/l
ogstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logst
ash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "/usr/sh
are/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_clie
nt/pool.rb:83:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logst
ash/outputs/elasticsearch/http_client.rb:359:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output
-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bund
le/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `cr
eate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/out
puts/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-ela
sticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundl
e/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `register'", "org/logsta
sh/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:
68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jrub
y/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/u
sr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-corcode here
type or paste cod][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::Converge
Result::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2023-10-10T15:08:26,598][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2023-10-10T15:08:26,726][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2023-10-10T15:08:26,735][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2023-10-10T15:08:26,766][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2023-10-10T15:08:26,879][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>1}
[2023-10-10T15:08:26,928][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
[2023-10-10T15:08:26,965][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Stop/pipeline_id:
.monitoring-logstash}
[2023-10-10T15:08:27,014][DEBUG][logstash.javapipeline    ] Closing inputs {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thre
ad:0x1fc570e3@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 sleep>"}
[2023-10-10T15:08:27,042][DEBUG][logstash.inputs.metrics  ] Stopping {:plugin=>"LogStash::Inputs::Metrics"}
[2023-10-10T15:08:27,051][DEBUG][logstash.inputs.metrics  ] Metrics input: stopped
[2023-10-10T15:08:27,053][DEBUG][logstash.javapipeline    ] Closed inputs {:pipeline_id=>".monitoring-logstash", :thread=>"#<Threa
d:0x1fc570e3@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 sleep>"}
[2023-10-10T15:08:27,139][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline.
[2023-10-10T15:08:27,609][DEBUG][logstash.inputs.metrics  ] Closing {:plugin=>"LogStash::Inputs::Metrics"}
[2023-10-10T15:08:27,687][DEBUG][logstash.pluginmetadata  ] Removing metadata for plugin fed3e7a74efd721e491df92d7ccb4632b4cecedfe
06f0513092b0ee4a70dc76c
[2023-10-10T15:08:27,748][DEBUG][logstash.javapipeline    ] Input plugins stopped! Will shutdown filter/output workers. {:pipeline
_id=>".monitoring-logstash", :thread=>"#<Thread:0x1fc570e3@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.r
b:54 run>"}
[2023-10-10T15:08:27,885][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>".monitoring-logstas
h", :thread=>"#<Thread:0x2982fec1 run>"}
[2023-10-10T15:08:28,025][DEBUG][logstash.outputs.elasticsearchmonitoring] Closing {:plugin=>"LogStash::Outputs::ElasticSearchMoni
toring"}
[2023-10-10T15:08:28,123][DEBUG][logstash.outputs.elasticsearchmonitoring] Stopping sniffer
[2023-10-10T15:08:28,173][DEBUG][logstash.outputs.elasticsearchmonitoring] Stopping resurrectionist
e here

Not sure if this helps..

Hi,

i tried to solve the previous issues.
But getting a new issue consistently. Can I know what this error means and some inputs ?

[2023-10-12T12:01:29,528][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.ba
tch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x5f610843 run>"}
[2023-10-12T12:01:35,165][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>5.6}
[2023-10-12T12:01:35,441][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>".monitoring-logstash"} 
[2023-10-12T12:01:35,976][ERROR][logstash.javapipeline    ] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>".monitoring-logstash", :error=>"(NoMethodError) un
defined method `shutdown_requested?' for #<LogStash::JavaPipeline:0x307ba0c7>", :exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>["usr.share.logstash.vendor.bundl
e.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_18_dot_0_minus_java.lib.logstash.plugin_mixins.elasticsearch.common.pipeline_shutdown_requested
?(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.18.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:386)", "usr.share.logstash.vend
or.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_18_dot_0_minus_java.lib.logstash.outputs.elasticsearch.wait_for_successful_connection(/
usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.18.0-java/lib/logstash/outputs/elasticsearch.rb:428)", "usr.share.logstash.vendor.bundle.jruby.
$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_11_dot_18_dot_0_minus_java.lib.logstash.outputs.elasticsearch.multi_receive(/usr/share/logstash/vendor/bundle/
jruby/2.5.0/gems/logstash-output-elasticsearch-11.18.0-java/lib/logstash/outputs/elasticsearch.rb:378)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategy
Ext.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:143)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/c
ompiler/AbstractOutputDelegatorExt.java:121)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/jav
a_pipeline.rb:299)"], :thread=>"#<Thread:0x5f610843 sleep>"}
[2023-10-12T12:01:36,126][WARN ][logstash.javapipeline    ] Waiting for input plugin to close {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x5f610843 run>"}
[2023-10-12T12:01:37,990][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2023-10-12T12:01:38,672][INFO ][logstash.runner          ] Logstash shut down.

Did you check the system logs as asked in a previous post?

Please check the system logs, /var/log/messages or /var/log/syslog for some hints why the service is not starting.

Also, can you share the logs from the beginning when logstash start? You didn't share all the logs when it start, you will have a couple more lines before the ones you shared.

Try the following, stop the service, remove the file /var/log/logstash/logstash-plain.log, start the service again to get fresh logs and share all the logs in the file.

Something is not right, if it was workign and suddenly stop working, so something was changed in the system or in logstash configuration.

Were you able to track this as you mentioned?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.