I am running an upgrade from some older versions of opendistro and OSS logstash to the latest versions, in order to mitigate some of the latest vulnerabilities.
According to Elastic, Logstash OSS 7.16.x should be compatible with Elasticsearch 7.10.x .
And according to OpenSearch/OpenDistro/AWS (Cool cats have many names), Open Distro Elasticsearch 1.13.3 should be running Elasticsearch 7.10.2 under the hood.
However, when I run up a cluster with Logstash-OSS 7.16.1 using an output pipeline to an Open Distro Elasticsearch 1.13.3, I'm getting the following incompatibility error in Logstash OSS:
[2021-12-17T08:12:02,583][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-12-17T08:12:02,598][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-12-17T08:12:02,608][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-12-17T08:12:02,618][INFO ][logstash.outputs.elasticsearch][output-elasticsearch_local] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://elkstack-node:9200"]}
[2021-12-17T08:12:02,639][INFO ][logstash.outputs.elasticsearch][output-elasticsearch_local] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://USER:xxxxxx@elkstack-node:9200/]}}
[2021-12-17T08:12:02,717][ERROR][logstash.javapipeline ][output-elasticsearch_local] **Pipeline error {:pipeline_id=>"output-elasticsearch_local", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>**, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/outputs/output-elasticsearch_local.logstash.conf"], :thread=>"#<Thread:0x368d0eba run>"}
[2021-12-17T08:12:02,723][INFO ][logstash.javapipeline ][output-elasticsearch_local] Pipeline terminated {"pipeline.id"=>"output-elasticsearch_local"}
[2021-12-17T08:12:02,729][ERROR][logstash.agent ] Failed to execute action {:id=>:"output-elasticsearch_local", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<output-elasticsearch_local>, action_result: false", :backtrace=>nil}
Output from Docker to verify versions:
someadmin@someplace:/var/git/Updated_OpenDistro_ElasticSearch/Docker$ docker ps --format "table {{.Names}}\t{{.Status}}\t{{.RunningFor}}\t{{.Image}}"
NAMES STATUS CREATED IMAGE
elkstack-kibana Up 45 minutes 45 minutes ago amazon/opendistro-for-elasticsearch-kibana:1.13.2
elkstack-logstash Up 45 minutes 45 minutes ago logstash-oss:7.16.1
elkstack-node Up 45 minutes 45 minutes ago amazon/opendistro-for-elasticsearch:1.13.3
My logstash output to Elasticsearch pipeline:
input {
pipeline {
address => "output-elasticsearch_local"
}
}
output {
elasticsearch {
hosts => ["${ELASTICSERVER:not_set}"]
ssl => true
cacert => "/usr/share/logstash/config/ca.pem"
ssl_certificate_verification => true
user => "${ELASTIC_LOGSTASH_USER:not_set}"
password => "${ELASTIC_LOGSTASH_USER_PASSWORD:not_set}"
ilm_enabled => false
index => "logstash-%{[@metadata][index_prefix]}"
}
}
The nodes can reach each-other, so the resolved variables in the above are correct. I have verified that as well manually, logstash is not reaching a wrong cluster, as it is an isolated environment.
I'm aware that this is crossing between an OpenDistro and Elastic problem, but i'm attacking it from both ends (posted in both forums) to figure out if i missed something in either.