# How to set allow\_insecure\_settings to enable repository-s3

**URL:** <https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [April 19, 2024, 7:33am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771 "2024-04-19T07:33:50Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [April 19, 2024, 7:33am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/1 "2024-04-19T07:33:50Z")

</div>

I installed elastic 8.13.0 in OCP 4.15. I attach to a elastic pod and tried to register a S3 repository to minio

```auto
sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repository?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "http://here-is-my-minio:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "base_path": "test-1",
> "access_key": "here is my key",
> "secret_key": "here is my secret"
> }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repository] path [test-1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository] path [test-1] is not accessible on master node",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Setting [access_key] is insecure, but property [allow_insecure_settings] is not set"
    }
  },
  "status" : 500
}

```

then I added `allow_insecure_settings: true` in `elasticsearch.yml` and restarted my elastic pods,

then my pod keep crashes. I get these error from pod logs.

```auto
{"@timestamp":"2024-04-19T07:31:33.210Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"instana-es-default-0","elasticsearch.cluster.name":"instana","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [April 19, 2024, 7:52am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/2 "2024-04-19T07:52:10Z")

</div>

I also tried an approach like this

```auto
echo "-Des.allow_insecure_settings=true" >> /usr/share/elasticsearch/config/jvm.options 

```

but when I reload security settings, it said like this

```auto
sh-5.0$ curl -X POST "http://instana-es-http:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "secure_settings_password":""
> }'
{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"instana","nodes":{"DP1kzLnKR2SvGG3xd4ThAQ":{"name":"instana-es-default-0","reload_exception":{"type":"exception","reason":"secure settings reload failed for one or more security components","suppressed":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}]}}}}sh-5.0$

```

---

<div class="post-metadata">

**Author:** ![Nabeel\_Ahmed\_NAK](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nabeel_ahmed_nak/32/133673_2.png) [@Nabeel\_Ahmed\_NAK](https://discuss.elastic.co/u/Nabeel_Ahmed_NAK)\
**Post date:** [April 19, 2024, 8:04am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/3 "2024-04-19T08:04:19Z")

</div>

> [@dove-young](#):
>
> `secret_key`

Hi @dove-young

Did you use elasticsearch-keystore to save secure settings?  
If not please use it to save access\_key, and secret\_key.

Thanks

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [April 21, 2024, 12:53pm UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/4 "2024-04-21T12:53:07Z")

</div>

> [@dove-young](#):
>
> then I added `allow_insecure_settings: true` in `elasticsearch.yml` and restarted my elastic pods,

Where did you see instructions to do that?  
I believe it needs to be a system property, not a configuration setting.

It would be better if you put the keys in the keystore instead.

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [April 22, 2024, 2:27am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/5 "2024-04-22T02:27:56Z")

</div>

you know, the problem is there is NO document for it. Do you think that there SHOULD be some document to tell what to do and how to do?

And I tried keystores and I also hit failures. My minio works and I can use it for velero backup and restore. I don't have problem on minio when I tried velero and tried `mc ls`

```auto
mc ls st2-tunnel-dev1/elastic-o1-726938/
[2024-04-21 19:44:24 PDT] 0B test-1/

```

Here is error when I tried to create a repository with keystore. The repository created but not correct

```auto
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:
sh-5.0$
sh-5.0$ curl -X POST "http://instana-es-http:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "secure_settings_password":""
> }'
{"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"instana","nodes":{"5j2gMbznQ32TBY0q_F7bhw":{"name":"instana-es-default-2"},"DXTs-bK9SNCtdZROE4OQpQ":{"name":"instana-es-default-1"},"cBoIh1OAQeydSARbilMqmQ":{"name":"instana-es-default-0"}}}sh-5.0$
sh-5.0$

sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo2?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "http://9.112.252.135:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "base_path": "test-1"
> }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo2] path [test-1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo2] path [test-1] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [test-1/tests-N96G-ulORKyP0D52JFU87w/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Connection refused"
        }
      }
    }
  },
  "status" : 500
}
sh-5.0$

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [April 22, 2024, 2:56am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/6 "2024-04-22T02:56:24Z")

</div>

And I can show my credential from keystore. My credential is masked in this post.

```auto
sh-5.0$ bin/elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

sh-5.0$ bin/elasticsearch-keystore show s3.client.default.access_key
my-key ## faked
sh-5.0$ bin/elasticsearch-keystore show s3.client.default.secret_key
my-secret ## faked
sh-5.0$

```

Also I tried to access Minio via curl in the elastic pod, the network endpoint is reachable. And I also tried to create repository without `base_path` param, the result is the same error.

```auto
sh-5.0$ curl http://9.112.252.135:9000
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><Region>eu-central-1</Region><RequestId>17C87B6D4950BB9B</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>sh-5.0$

```

I changed access mode of this bucket to `public` in minio and it did not fix this problem

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [April 22, 2024, 4:10am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/7 "2024-04-22T04:10:42Z")

</div>

I changed to Elasticsearch v8.13.2, and I tried to run a minio server on the infrastructure node in the openshift cluster to try to filter out any potential network problem. But I found it become more interesting.

When I use domain name to access minio and use IP address to access minio, Elasticsearch API replies are totally different.

Here is when I use domain name to access minio, elasticsearch use bucket name as a subdomain name to make http request.

```auto
sh-5.0$
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.access_key
Setting s3.client.default.access_key already exists. Overwrite? [y/N]y
Enter value for s3.client.default.access_key:
sh-5.0$ bin/elasticsearch-keystore add s3.client.default.secret_key
Setting s3.client.default.secret_key already exists. Overwrite? [y/N]y
Enter value for s3.client.default.secret_key:
sh-5.0$
sh-5.0$
sh-5.0$ curl -X POST "http://instana-es-http:9200/_nodes/reload_secure_settings" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "secure_settings_password":""
> }'
{"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"instana","nodes":{"KwRBhcaSQTao6rWZJc_YIQ":{"name":"instana-es-default-2"},"tkOsorSSQ9mVvyPIf7Z6lQ":{"name":"instana-es-default-0"},"c_4d50bQQd6Hz8simnAUrg":{"name":"instana-es-default-1"}}}sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo8?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "http://api.o1-729588.my-domain:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "base_path": "test-1"
> }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo8] path [test-1] is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo8] path [test-1] is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [test-1/tests-191aIWDSSbi0u2EHKzS92w/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: Unable to execute HTTP request: elastic-o1-726938.api.o1-729588.my-domain",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "elastic-o1-726938.api.o1-729588.my-domain"
        }
      }
    }
  },
  "status" : 500
}
sh-5.0$ curl elastic-o1-726938.api.o1-729588.my-domain
curl: (6) Could not resolve host: elastic-o1-726938.api.o1-729588.my-domain
sh-5.0$
sh-5.0$

```

---

<div class="post-metadata">

**Author:** ![dove-young](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dove-young/32/133625_2.png) [@dove-young](https://discuss.elastic.co/u/dove-young)\
**Post date:** [April 22, 2024, 4:12am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/8 "2024-04-22T04:12:43Z")

</div>

when I used IP address to access minio server, the elasticsearch API reply message are totally different.

```auto
sh-5.0$ curl -X PUT "http://instana-es-http:9200/_snapshot/my_s3_repo9?pretty" -H 'Content-Type: application/json' -u elastic:$ELASTIC_PASSWORD -d'
> {
> "type": "s3",
> "settings": {
> "endpoint": "http://9.46.195.49:9000",
> "protocol": "http",
> "region": "eu-central-1",
> "bucket": "elastic-o1-726938",
> "base_path": "test-1"
> }
> }'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo9] [[c_4d50bQQd6Hz8simnAUrg, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-1][10.254.24.35:9300][internal:admin/repository/verify]'], [KwRBhcaSQT
ao6rWZJc_YIQ, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-2][10.254.12.60:9300][internal:admin/repository/verify]']]",
        "suppressed" : [
          {
            "type" : "repository_verification_exception",
            "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-1}{c_4d50bQQd6Hz8simnAUrg}{ikqTtjcWTsewflAvH_1sAQ}{instana-es-default-1}{10.254.2
4.35}{10.254.24.35:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0, k8s_node_name=worker5.o1-729588.my-domain,
transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm_size=1073741824}]"
          },
          {
            "type" : "repository_verification_exception",
            "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-2}{KwRBhcaSQTao6rWZJc_YIQ}{E92hTrYETDuFv70fm7qg7Q}{instana-es-default-2}{10.254.1
2.60}{10.254.12.60:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{k8s_node_name=worker0.o1-729588.my-domain, transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm
_size=1073741824, ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0}]"
          }
        ]
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repo9] [[c_4d50bQQd6Hz8simnAUrg, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-1][10.254.24.35:9300][internal:admin/repository/verify]'], [KwRBhcaSQTao6r
WZJc_YIQ, 'org.elasticsearch.transport.RemoteTransportException: [instana-es-default-2][10.254.12.60:9300][internal:admin/repository/verify]']]",
    "suppressed" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-1}{c_4d50bQQd6Hz8simnAUrg}{ikqTtjcWTsewflAvH_1sAQ}{instana-es-default-1}{10.254.24.35
}{10.254.24.35:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0, k8s_node_name=worker5.o1-729588.my-domain, tran
sform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm_size=1073741824}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [test-1/tests-IWXppL4XR1m2w7n4akvb_Q/data-c_4d50bQQd6Hz8simnAUrg.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Connection refused"
            }
          }
        }
      },
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repo9] store location [elastic-o1-726938] is not accessible on the node [{instana-es-default-2}{KwRBhcaSQTao6rWZJc_YIQ}{E92hTrYETDuFv70fm7qg7Q}{instana-es-default-2}{10.254.12.60
}{10.254.12.60:9300}{cdfhilmrstw}{8.13.2}{7000099-8503000}{k8s_node_name=worker0.o1-729588.my-domain, transform.config_version=10.0.0, xpack.installed=true, ml.allocated_processors=16, ml.max_jvm_siz
e=1073741824, ml.config_version=12.0.0, ml.machine_memory=2147483648, ml.allocated_processors_double=16.0}]",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unable to upload object [test-1/tests-IWXppL4XR1m2w7n4akvb_Q/data-KwRBhcaSQTao6rWZJc_YIQ.dat] using a single upload",
          "caused_by" : {
            "type" : "sdk_client_exception",
            "reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
            "caused_by" : {
              "type" : "i_o_exception",
              "reason" : "Connection refused"
            }
          }
        }
      }
    ]
  },
  "status" : 500
}
sh-5.0$

```

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [April 22, 2024, 6:42am UTC](https://discuss.elastic.co/t/how-to-set-allow-insecure-settings-to-enable-repository-s3/357771/9 "2024-04-22T06:42:29Z")

</div>

> [@dove-young](#):
>
> Do you think that there SHOULD be some document to tell what to do and how to do?

No, because the documented, supported option is to use the keystore.

> [@dove-young](#):
>
> ```auto
> "caused_by" : {
> "type" : "i_o_exception",
> "reason" : "Connection refused"
> }
>  
> 
> ```

That's a network level error. It has nothing to do with access keys.

It does not look like it is possible to connect to `http://9.112.252.135:9000` from the master node in your cluster.

> [@dove-young](#):
>
> Also I tried to access Minio via curl in the elastic pod, the network endpoint is reachable. And I also tried to create repository without `base_path` param, the result is the same error.

I can't explain that. Are you absolutely sure you were inside the same pod that the master node is running on?  
ES isn't doing anything special here - if you can connect from a shell on the same host then there's no obvious reason why ES would get a connection failure.