Elasticsearch output - ILM availablity not properly detected with expired ES license

Hi community,

I think I discovered a corner case bug in the elasticsearch output plugin. Currently the license of my elasticsearch cluster has expired and it feels like the ILM availability is no longer properly detected by the output plugin.

My config looks like this:

output {
  elasticsearch {
    hosts => [ "host1", "host2" ]
    ilm_rollover_alias => "my-specific-alias"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "specific_alias_lifecycle"
    user => "user"
    password => "password"
  }
}

I know that the lifecycle feature is not disabled due to the expired license, because my indices still get cycled through. After restarting one of my logstash nodes though, this one is not able to detect ILM feature and therefore ships my data to the default logstash-* index instead. In the logs I see this:

New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["host1", "host2"]}
Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not available in your Elasticsearch cluster
Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{...}}}

Does anybody ever have seen this? Is it maybe a bug in logstash-output-elasticsearch/ilm.rb at master · logstash-plugins/logstash-output-elasticsearch · GitHub?

It's not a big issue for myself, since I will refresh my license in the next days, but with ILM being a feature, which is not degraded by an expired license, this is at least unexpected.

Best regards,
Sebastian

The elasticsearch output makes a request to ES for "/_xpack", which it assumes contains JSON. It expects both the 'available' and 'enabled' fields to be 'true' in that JSON. If ES does not set them to true when a licence expires I think that would be an ES bug.

Well, then this is the issue. I get the following JSON back

{
  "build":{"hash":"foo","date":"timestamp"},
  "license":{"uid":"bar","type":"platinum","mode":"platinum","status":"expired","expiry_date_in_millis":123456890},
  "features":{
    "analytics":{"available":false,"enabled":true},
    "ccr":{"available":false,"enabled":true},
    "data_streams":{"available":true,"enabled":true},
    "enrich":{"available":false,"enabled":true},
    "eql":{"available":false,"enabled":true},
    "flattened":{"available":false,"enabled":true},
    "frozen_indices":{"available":true,"enabled":true},
    "graph":{"available":false,"enabled":true},
    "ilm":{"available":false,"enabled":true},
    "logstash":{"available":false,"enabled":true},
    "ml":{"available":false,"enabled":true,"native_code_info":{"version":"7.9.1","build_hash":"abcdef"}},
    "monitoring":{"available":false,"enabled":true},
    "rollup":{"available":false,"enabled":true},
    "security":{"available":true,"enabled":true},
    "slm":{"available":false,"enabled":true},
    "spatial":{"available":false,"enabled":true},
    "sql":{"available":false,"enabled":true},
    "transform":{"available":false,"enabled":true},
    "vectors":{"available":false,"enabled":true},
    "voting_only":{"available":false,"enabled":true},
    "watcher":{"available":false,"enabled":false}
  },
  "tagline":"You know, for X"
}

This strikes me odd though. As far as I can tell from License expiration, it is not something, which should be degraded.

And the feature itself certainly is running. My indices still properly cycle through their intended lifecycle.

It may just be an elasticsearch bug. You might get an answer if you post that JSON in the elasticsearch forum and ask why available is false for ilm when the licence is expired.