How to set allow_insecure_settings to enable repository-s3

Elastic Stack Elasticsearch
1

I installed elastic 8.13.0 in OCP 4.15. I attach to a elastic pod and tried to register a S3 repository to minio

sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repository?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "http://here-is-my-minio:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "base_path": "test-1",
>     "access_key": "here is my key",
>     "secret_key": "here is my secret"
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repository] path [test-1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository] path [test-1] is not accessible on master node",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Setting [access_key] is insecure, but property [allow_insecure_settings] is not set"
    }
  },
  "status" : 500
}

then I added allow_insecure_settings: true in elasticsearch.yml and restarted my elastic pods,

then my pod keep crashes. I get these error from pod logs.

{"@timestamp":"2024-04-19T07:31:33.210Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
2

I also tried an approach like this

echo "-Des.allow_insecure_settings=true" >> /usr/share/elasticsearch/config/jvm.options

but when I reload security settings, it said like this

sh-5.0$ curl -X POST "http://instana-es-http:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "secure_settings_password":""
> }'
{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"instana","nodes":{"DP1kzLnKR2SvGG3xd4ThAQ":{"name":"instana-es-default-0","reload_exception":{"type":"exception","reason":"secure settings reload failed for one or more security components","suppressed":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}]}}}}sh-5.0$
3

Hi @dove-young

Did you use elasticsearch-keystore to save secure settings?
If not please use it to save access_key, and secret_key.

Thanks

4

Where did you see instructions to do that?
I believe it needs to be a system property, not a configuration setting.

It would be better if you put the keys in the keystore instead.

5

you know, the problem is there is NO document for it. Do you think that there SHOULD be some document to tell what to do and how to do?

And I tried keystores and I also hit failures. My minio works and I can use it for velero backup and restore. I don't have problem on minio when I tried velero and tried mc ls

mc  ls  st2-tunnel-dev1/elastic-o1-726938/
[2024-04-21 19:44:24 PDT]     0B test-1/

Here is error when I tried to create a repository with keystore. The repository created but not correct

sh-5.0$ bin/elasticsearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:
sh-5.0$
sh-5.0$ curl -X POST "http://instana-es-http:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "secure_settings_password":""
> }'
{"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"instana","nodes":{"5j2gMbznQ32TBY0q_F7bhw":{"name":"instana-es-default-2"},"DXTs-bK9SNCtdZROE4OQpQ":{"name":"instana-es-default-1"},"cBoIh1OAQeydSARbilMqmQ":{"name":"instana-es-default-0"}}}sh-5.0$
sh-5.0$

sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo2?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "http://9.112.252.135:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "base_path": "test-1"
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo2] path [test-1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo2] path [test-1] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [test-1/tests-N96G-ulORKyP0D52JFU87w/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}
sh-5.0$
6

And I can show my credential from keystore. My credential is masked in this post.

sh-5.0$ bin/elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

sh-5.0$ bin/elasticsearch-keystore show s3.client.default.access_key
my-key ## faked
sh-5.0$ bin/elasticsearch-keystore show s3.client.default.secret_key
my-secret ## faked
sh-5.0$

Also I tried to access Minio via curl in the elastic pod, the network endpoint is reachable. And I also tried to create repository without base_path param, the result is the same error.

sh-5.0$ curl http://9.112.252.135:9000
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><Region>eu-central-1</Region><RequestId>17C87B6D4950BB9B</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>sh-5.0$

I changed access mode of this bucket to public in minio and it did not fix this problem

7

I changed to Elasticsearch v8.13.2, and I tried to run a minio server on the infrastructure node in the openshift cluster to try to filter out any potential network problem. But I found it become more interesting.

When I use domain name to access minio and use IP address to access minio, Elasticsearch API replies are totally different.

Here is when I use domain name to access minio, elasticsearch use bucket name as a subdomain name to make http request.

sh-5.0$
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.access_key
Setting s3.client.default.access_key already exists. Overwrite? [y/N]y
Enter value for s3.client.default.access_key:
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.secret_key
Setting s3.client.default.secret_key already exists. Overwrite? [y/N]y
Enter value for s3.client.default.secret_key:
sh-5.0$
sh-5.0$
sh-5.0$ curl -X POST "http://instana-es-http:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "secure_settings_password":""
> }'
{"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"instana","nodes":{"KwRBhcaSQTao6rWZJc_YIQ":{"name":"instana-es-default-2"},"tkOsorSSQ9mVvyPIf7Z6lQ":{"name":"instana-es-default-0"},"c_4d50bQQd6Hz8simnAUrg":{"name":"instana-es-default-1"}}}sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo8?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "http://api.o1-729588.my-domain:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "base_path": "test-1"
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo8] path [test-1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo8] path [test-1] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [test-1/tests-191aIWDSSbi0u2EHKzS92w/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Unable to execute HTTP request: elastic-o1-726938.api.o1-729588.my-domain",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "elastic-o1-726938.api.o1-729588.my-domain"
        }
      }
    }
  },
  "status" : 500
}
sh-5.0$ curl elastic-o1-726938.api.o1-729588.my-domain
curl: (6) Could not resolve host: elastic-o1-726938.api.o1-729588.my-domain
sh-5.0$
sh-5.0$
8

when I used IP address to access minio server, the elasticsearch API reply message are totally different.

sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo9?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
>   "type": "s3",
>   "settings": {
>     "endpoint": "http://9.46.195.49:9000",
>     "protocol": "http",
>     "region": "eu-central-1",
>     "bucket": "elastic-o1-726938",
>     "base_path": "test-1"
>   }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo9] [[c_4d50bQQd6Hz8simnAUrg, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-1][10.254.24.35:9300][internal:admin/repository/verify]'], [KwRBhcaSQT
ao6rWZJc_YIQ, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-2][10.254.12.60:9300][internal:admin/repository/verify]']]",
        "suppressed" : [
          {
            "type" : "repository_verification_exception",
            "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-1}{c_4d50bQQd6Hz8simnAUrg}{ikqTtjcWTsewflAvH_1sAQ}{instana-es-default-1}{10.254.2
4.35}{10.254.24.35:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0, k8s_node_name=worker5.o1-729588.my-domain,
transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm_size=1073741824}]"
          },
          {
            "type" : "repository_verification_exception",
            "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-2}{KwRBhcaSQTao6rWZJc_YIQ}{E92hTrYETDuFv70fm7qg7Q}{instana-es-default-2}{10.254.1
2.60}{10.254.12.60:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{k8s_node_name=worker0.o1-729588.my-domain, transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm
_size=1073741824, ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0}]"
          }
        ]
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo9] [[c_4d50bQQd6Hz8simnAUrg, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-1][10.254.24.35:9300][internal:admin/repository/verify]'], [KwRBhcaSQTao6r
WZJc_YIQ, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-2][10.254.12.60:9300][internal:admin/repository/verify]']]",
    "suppressed" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-1}{c_4d50bQQd6Hz8simnAUrg}{ikqTtjcWTsewflAvH_1sAQ}{instana-es-default-1}{10.254.24.35
}{10.254.24.35:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0, k8s_node_name=worker5.o1-729588.my-domain, tran
sform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm_size=1073741824}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [test-1/tests-IWXppL4XR1m2w7n4akvb_Q/data-c_4d50bQQd6Hz8simnAUrg.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Connection refused"
            }
          }
        }
      },
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-2}{KwRBhcaSQTao6rWZJc_YIQ}{E92hTrYETDuFv70fm7qg7Q}{instana-es-default-2}{10.254.12.60
}{10.254.12.60:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{k8s_node_name=worker0.o1-729588.my-domain, transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm_siz
e=1073741824, ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [test-1/tests-IWXppL4XR1m2w7n4akvb_Q/data-KwRBhcaSQTao6rWZJc_YIQ.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Connection refused"
            }
          }
        }
      }
    ]
  },
  "status" : 500
}
sh-5.0$
9

No, because the documented, supported option is to use the keystore.

That's a network level error. It has nothing to do with access keys.

It does not look like it is possible to connect to http://9.112.252.135:9000 from the master node in your cluster.

I can't explain that. Are you absolutely sure you were inside the same pod that the master node is running on?
ES isn't doing anything special here - if you can connect from a shell on the same host then there's no obvious reason why ES would get a connection failure.