Problems with Logstash Pipeline to ElasticSearch

Elastic Stack Logstash
1

Hi i have some erros to up my logstash service as bellows

[2021-12-03T19:44:46,535][INFO ][logstash.javapipeline    ][KSC] Pipeline terminated {"pipeline.id"=>"KSC"}
[2021-12-03T19:44:46,550][ERROR][logstash.agent           ] Failed to execute action {:id=>:Forti_IPS, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<Forti_IPS>, action_result: false", :backtrace=>nil}
[2021-12-03T19:44:46,551][ERROR][logstash.agent           ] Failed to execute action {:id=>:KSC, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<KSC>, action_result: false", :backtrace=>nil}
[2021-12-03T19:44:46,799][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-12-03T19:44:51,692][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-03T19:44:51,767][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
2

pipelines.yml

# cat pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: Forti_IPS
  path.config: "/etc/logstash/conf.d/forti_IPS.conf"

- pipeline.id: KSC
  path.config: "/etc/logstash/conf.d/ksc.conf"
3

If you enable log.level debug do you get any more informative messages?

4

Follows

[2021-12-03T20:20:29,000][INFO ][logstash.javapipeline    ][Forti_IPS] Pipeline terminated {"pipeline.id"=>"Forti_IPS"}
[2021-12-03T20:20:29,023][ERROR][logstash.agent           ] Failed to execute action {:id=>:Forti_IPS, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<Forti_IPS>, action_result: false", :backtrace=>nil}
[2021-12-03T20:20:29,062][DEBUG][logstash.agent           ] Starting puma
[2021-12-03T20:20:29,088][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2021-12-03T20:20:29,089][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2021-12-03T20:20:29,112][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2021-12-03T20:20:29,116][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2021-12-03T20:20:29,120][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2021-12-03T20:20:29,129][DEBUG][logstash.api.service     ] [api-service] start
[2021-12-03T20:20:29,132][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2021-12-03T20:20:29,142][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2021-12-03T20:20:29,278][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-12-03T20:20:34,363][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-03T20:20:34,389][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
5

Can you add few lines before [2021-12-03T20:20:29,000][INFO ][logstash.javapipeline ][Forti_IPS] Pipeline ...

6

You need to share your pipelines configuration as well to help understand why logstash can't start them.

7

I`m already share, if you see in the earlier posts.

8 
filter_to_output -> P[output-stdout{"codec"=>"rubydebug"}|[file]/etc/logstash/conf.d/forti_IPS.conf:46:1:```
stdout { codec => rubydebug }
```]
**GRAPH**
[2021-12-04T10:07:36,916][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"Forti_IPS"}
[2021-12-04T10:07:36,973][DEBUG][logstash.outputs.elasticsearch][Forti_IPS] Normalizing http path {:path=>nil, :normalized=>nil}
[2021-12-04T10:07:37,000][WARN ][logstash.outputs.elasticsearch][Forti_IPS] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2021-12-04T10:07:37,428][INFO ][logstash.outputs.elasticsearch][Forti_IPS] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_scbraganca:xxxxxx@glb-elkdat-01:9200/]}}
[2021-12-04T10:07:37,477][DEBUG][logstash.outputs.elasticsearch][Forti_IPS] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_scbraganca:xxxxxx@glb-elkdat-01:9200/, :path=>"/"}
[2021-12-04T10:07:37,907][DEBUG][logstash.outputs.stdout  ][Forti_IPS] Closing {:plugin=>"LogStash::Outputs::Stdout"}
[2021-12-04T10:07:37,917][DEBUG][logstash.pluginmetadata  ][Forti_IPS] Removing metadata for plugin 39b3c9a14b210b713f4418afb2053ce3066e93d41d03732550f8c92080515d19
[2021-12-04T10:07:37,923][ERROR][logstash.javapipeline    ][Forti_IPS] Pipeline error {:pipeline_id=>"Forti_IPS", :exception=>#<Manticore::UnknownException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:37:in `block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.7.0-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:74:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:261:in `health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:270:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:266:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:382:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:82:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:76:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:302:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:64:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:105:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:101:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch.rb:307:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.3-java/lib/logstash/outputs/elasticsearch/common.rb:23:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:585:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/forti_IPS.conf"], :thread=>"#<Thread:0x2ad13271 run>"}
[2021-12-04T10:07:37,930][INFO ][logstash.javapipeline    ][Forti_IPS] Pipeline terminated {"pipeline.id"=>"Forti_IPS"}
[2021-12-04T10:07:37,950][ERROR][logstash.agent           ] Failed to execute action {:id=>:Forti_IPS, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<Forti_IPS>, action_result: false", :backtrace=>nil}
[2021-12-04T10:07:37,999][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2021-12-04T10:07:38,007][DEBUG][logstash.agent           ] Starting puma
[2021-12-04T10:07:38,021][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2021-12-04T10:07:38,035][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2021-12-04T10:07:38,037][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2021-12-04T10:07:38,039][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2021-12-04T10:07:38,055][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2021-12-04T10:07:38,061][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2021-12-04T10:07:38,096][DEBUG][logstash.api.service     ] [api-service] start
[2021-12-04T10:07:38,227][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-12-04T10:07:43,101][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-04T10:07:43,155][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2021-12-04T10:07:43,164][DEBUG][logstash.agent           ] 2021-12-04 10:07:43 -0300: Listen loop error: #<IOError: closed stream>
org/jruby/RubyIO.java:3067:in `read'
org/jruby/RubyIO.java:3049:in `read'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.7-java/lib/puma/server.rb:440:in `handle_check'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.7-java/lib/puma/server.rb:386:in `block in handle_servers'
org/jruby/RubyArray.java:1809:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.7-java/lib/puma/server.rb:384:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.7-java/lib/puma/server.rb:356:in `block in run'
9 
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: Fortigate_IPS
  path.config: "/etc/logstash/conf.d/forti_IPS.conf"

- pipeline.id: KSC
  path.config: "/etc/logstash/conf.d/ksc.conf"
10

:exception=>#<Manticore::UnknownException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty>

The JVM failed to read the trust store. This SO Q&A has some pointers.

11

This is your pipelines.yml, the pipelines configuration are the content of the .conf files.

It would help to know what is the content of the forti_IPS.conf and ksc.conf, maybe there is something wrong in the configuration that can give a hint of why logstash cannot start.

12

Hello,

After some time, I reinstall the whole enviroment (linux, logstash package, conf files), and put a configuration files of other logstash that's already running fine!

I see, if I change my logstash output certificate to connect in my Elasticsearch cluster, appears the error bellow

[2021-12-10T16:37:25,267][WARN ][logstash.outputs.elasticsearch][KSC] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_fccsa:xxxxxx@glb-elkdat-01:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_fccsa:xxxxxx@glb-elkdat-01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[2021-12-10T16:37:25,338][WARN ][logstash.outputs.elasticsearch][Forti_IPS] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_fccsa:xxxxxx@glb-elkdat-01:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_fccsa:xxxxxx@glb-elkdat-01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

So I belive the problem is in the command-line that I use to generate this certificate on my Elasticsearch master node, follows the cli I used.

/usr/share/Elasticsearch/bin/Elasticsearch-certutil cert -multiple --ca elastic-stack-ca.p12

I tried # bin/Elasticsearch-certutil cert --keep-ca-key --multiple --pem command too, not works

in the logstash side, In the case of pkcs12 certificate, I convert them to .pem file running:

openssl pkcs12 -in proxy-scbraganca.p12 -out proxy-scbraganca.crt.pem -clcerts -nokeys

So, what I'm doing wrong with this certificate generation?

13

Follow an output example of this pipeline configuration.

output {
elasticsearch {
hosts => ["https://hostname-of-elasticsearch:9200"]
#ssl => true
ssl_certificate_verification => false
cacert => "/opt/certs/certifitcate-crt.pem"
#ilm_enabled => false
user => logstash_fccsa
password => password
index => "ksc_index-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.