I have multiple pipelines running in my logstash server that have been configured via kibana. However, I'm facing an issue where if one of the pipeline configurations has some error than the entire logstash instance aborts and the logstash service stops. How do I ensure that the logstash instance does not go down and the remaining (error-free) pipelines keep running even if any of the other pipeline's encounter an error?
How are you deploying the configuration files? Before putting new files in place you can start Logstash with the config test flag to have it validate the configuration. If that test passes you can proceed with the file installation.
Hi Magnus,
We are configuring the pipeline through Kibana -> Management -> Pipelines and specifying the pipeline ID in logstash.yml.
The config test flag that you speak of is used through command line when we are loading the pipeline file through command line right? Is there anything similar that could be enabled in the logstash.yml file or any other logstash setting files?
The config test flag that you speak of is used through command line when we are loading the pipeline file through command line right?
Yes.
Is there anything similar that could be enabled in the logstash.yml file or any other logstash setting files?
Not as far as I know.
The config test flag can be set in the logstash.yml through the setting config.test_and_exit:true.
But I am encountering an error while trying to implement this. I will try to explore further and check if this solution can be used for my issue.
However, I'm quite confused why does logstash abort due to a single pipeline configuration error. This blocks the execution of the remaining pipelines that are properly configured.
I am unable to set the config.test_and_exit:true as we have centralized pipeline management enabled. Thus it returns the error message:
[logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/opt/appl/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.3-java/modules/arcsight/configuration"}
ERROR: You cannot use -t since Elasticsearch is configured as the config store
IG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
I was wondering if there is any alternative to configure this setting?
None that I know of.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.