Logstash crashes out with the following message

Elastic Stack Logstash
1

Logstash crashes out with the following log message:

PS C:\logstash\bin> .\logstash.bat
"Using bundled JDK: ."
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in
a future release.
Sending Logstash logs to C:\logstash\logs which is now configured via log4j2.properties
[2022-03-28T16:12:11,898][INFO ][logstash.runner          ] Log4j configuration path used is: C:\logstash\config\log4j2.
properties
[2022-03-28T16:12:11,907][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.1", "jruby.vers
ion"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-
x86_64]"}
[2022-03-28T16:12:12,056][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or
 command line options are specified
[2022-03-28T16:12:13,483][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ss
l_enabled=>false}
[2022-03-28T16:12:18,142][INFO ][org.reflections.Reflections] Reflections took 77 ms to scan 1 urls, producing 119 keys
and 417 values
[2022-03-28T16:12:20,223][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outp
uts::ElasticSearch", :hosts=>["https://bimbeats.es.eu-west-3.aws.elastic-cloud.com:9243"]}
[2022-03-28T16:12:20,513][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:remo
ved=>[], :added=>[https://logstash:xxxxxx@bimbeats.es.eu-west-3.aws.elastic-cloud.com:9243/]}}
[2022-03-28T16:12:20,926][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Lo
gStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["C:/logstash/vendo
r/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.
rb:247:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "C:/logstash/vendor/bundle/jruby/2.5.0/gem
s/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `healthcheck!'
", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsea
rch/http_client/pool.rb:374:in `update_urls'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch
-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "C:/logstash/vendor/bu
ndle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:8
3:in `start'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/output
s/elasticsearch/http_client.rb:359:in `build_pool'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elastic
search-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "C:/logstash/vendor/bundle/jru
by/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in
 `create_http_client'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logsta
sh/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-ou
tput-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "C:/logstash/ve
ndor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `reg
ister'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/Ab
stractOutputDelegatorExt.java:68:in `register'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block
in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.
rb:231:in `register_plugins'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'
", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "C:/logstash/logstash-core/lib/logs
tash/java_pipeline.rb:189:in `run'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"],
 "pipeline.sources"=>["C:/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x320c5ca0 run>"}
[2022-03-28T16:12:20,929][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-03-28T16:12:20,947][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash
::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false"
, :backtrace=>nil}
[2022-03-28T16:12:21,020][INFO ][logstash.runner          ] Logstash shut down.
[2022-03-28T16:12:21,029][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit
) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at C_3a_.logstash.lib.bootstrap.environment.<main>(C:\logstash\lib\bootstrap\environment.rb:94) ~[?:?]
PS C:\logstash\bin>

Any ideas on how to address this would be appreciated. Just for context. It's an official Logstash version 7.16.1, trying to connect to Elastic Cloud hosted database that was deployed to version 7.17.1.

2

This thread addresses that. I would suggest hitting the root elasticsearch URL and working out what the code will do with the JSON that is returned.

3

By hitting you mean pinging?

4

I mean doing a "GET /" using curl or a web browser.

5

I tried that and I am getting the following message:

StatusCode        : 200
StatusDescription : OK
Content           : {
                      "name" : "instance-0000000005",
                      "cluster_name" : "c2da7e382f20448c8",
                      "cluster_uuid" : "ln_4-tePQ5yagPD",
                      "version" : {
                        "number" : "7.17.1",
                        "build_flavor" : "d…
RawContent        : HTTP/1.1 200 OK
                    X-Cloud-Request-Id: SG-VoxlMIVintg
                    X-Elastic-Product: Elasticsearch
                    X-Found-Handling-Cluster: 8c800c8d53a577f2d7
                    X-Found-Handling-Instance: instance-000000000…
Headers           : {[X-Cloud-Request-Id, System.String[]], [X-Elastic-Product, System.String[]],
                    [X-Found-Handling-Cluster, System.String[]], [X-Found-Handling-Instance, System.String[]]…}
Images            : {}
InputFields       : {}
Links             : {}
RawContentLength  : 571
RelationLink      : {}
6

That's not valid JSON. If Elasticsearch is really returning that it is incredibly broken. Otherwise, the path through the code depends on the exact contents of "Content".

7

@Badger when you say that this JSON is invalid and Elasticsearch instance is "broken" how is it possible that I can stand up Logstash on a different computer/my personal machine, and it is able to connect to that exact same Elasticsearch instance and it works just fine? Thoughts? I am also able to post data to it via NEST library in .NET.

8

For now, I have re-created the whole deployment. It looks like I can connect to it from Logstash...for now. This is so strange. I will circle back if it stops working again.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.