Log stash not connecting to Elasticsearch

Elastic Stack Logstash
#1

Hi Team,
my log stash is unable to connect Elasticsearch. Telnet is working fine but log stash pipeline is not able to create connection with Elasticsearch.

input {
    stdin{}
}




output {
elasticsearch {
hosts => ["10.128.0.3:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}

Error message -

javapipeline - Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client/pool.rb:245:in block in healthcheck!'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client/pool.rb:238:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:370:in update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client/pool.rb:87:in update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:81:in start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client.rb:358:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/outputs/Elasticsearch/http_client_builder.rb:106:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-Elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/Elasticsearch/common.rb:34:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:279:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:598:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/beats.conf"], :thread=>"#<Thread:0x6ed41164 run>"}

Log stash pipeline not connecting to Elasticsearch
#2

Which Elasticsearch are you using? This normally happens when you are using the AWS Elasticsearch distribution with a newer Logstash version.

Also, what are the version of your Elasticsearch and Logstash?

#3

HI Leandrojmp , Thanks for your response.

Elasticsearch - 7.10.2

Logstash - 7.17.4

[root@master-red hariuser]# curl -X GET http://localhost:9200
{
  "name" : "master",
  "cluster_name" : "my-application",
  "cluster_uuid" : "samV8dK-ReWWZhESBwXmLQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

log stash -

 runner - Starting Logstash {"logstash.version"=>"7.17.4", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86_64]"}

How can i check whether it is AWS Elasticsearch distribution with a newer Logstash version or not ?

#4

This is the issue, you are running the open source version, you need to use Logstash in the same version, 7.10.2.

Newer versions of Logstash will check if you are using an Elasticsearch distributed by Elastic, using the basic free license or one of the paid tiers, the OSS version does not have the _license endpoint, so Logstash will not work.

Change your Logstash to 7.10.2 and it should work.

1 Like
#5

HI Leandrojmp , Now it is resolved. Thanks for your quick response.