Secure Settings Keystore vs. Config File

[mattdonders]

We are running ECE 1.1.1 and are facing an issue regarding the secure settings moved from the configuration to the keystore in ES5.5x and above. We hit these issues when migrating a cluster (from ES5.6.4 to ES6.0.0) or when creating a new ES6.0.0 cluster. Our assumption is that the ES5.6.4 cluster can be created due to the fact that this version deprecates these secure settings, but does not remove (or ignore) them.

Based on the information provided in this Github Issue (which I have left a comment on as well), this shouldn't be an issue for a new ES6 cluster, but we are facing the issue here as well. We believe we are failing for snapshots based on the HEAD for AWS endpoint <bucket>.s3.amazonaws.com since we are running Minio (an S3-work-a-like repository) internally.

We are also observing that if the snapshot repository cannot be connected to (or found), the cluster cannot be stopped unless snapshots are disabled before stopping cluster.

Instead of manually updating the items in the keystore after cluster creation, is there a way to at least use the configuration to create the keystore when a new cluster is created?

I have added our ES6 cluster configuration below (with some information obfuscated) as well as a screenshot from the startup of a new ES6 cluster in ECE 1.1.1 to this for review.

{
  "type": "s3",
  "settings": {
    "access_key": "XXXXXXXXXXXXXXXXXXXXXXX",
    "secret_key": "+XXXXXXXXXXXXXXXXXXXXX/XXXXXXXXX",
    "bucket": "snapshots",
    "region": "us-east-local",
    "endpoint": "http://172.19.xx.xxx:9000",
    "protocol": "http"
  }

Screenshot of Errors

}

[Alex_Piggott]

Hi @mattdonders

This is a known issue - I believe the only known workaround currently is to add the snapshot repo to a new cluster, apologies. A number of users have run into the same problem

We will release a fix for this in the next minor version (which doesn't yet have a timescale associated with it)

(One possible workaround would be to hand-edit the temporary /app/config/elasticsearch.yaml files in the containers/restart ES and then reapply the plan ... I think we'd recommend creating a new cluster with the right settings and a clone of the data though)