What Elasticsearch privileges should Curator have?

Elasticsearch version: 6.8.2
Elasticsearch Curator version: 5.7.6

My action files were working fine before I enabled TLS.

So in order to make sure, I spun up 2 instances of ELK (1 with and 1 without TLS) using helm and Kubernetes. Everything is identical except the extra secret volume mount and some security settings.

config.yml:

config.yml: |-
  ---
  client:
    hosts:
      - 127.0.0.1
    port: 9200
    # url_prefix:
    use_ssl: True # If set to True, curator will connect with HTTPS instead of HTTP
    certificate: /etc/es-certs/elastic-certificates.p12 # Absolute path to the root CA
    # client_cert and client_key are used for Client Certificate Authentication
    # client_cert: /etc/elasticsearch/config/client_certificate.pem # Absolute path to the client TLS certificate that is sent with each request.
    # client_key: /etc/elasticsearch/config/private_key.pem # Absolute path to the private key of the client certificate.
    ssl_no_validate: True # If set to True curator will not validate the certificate it receives from Elasticsearch. Enable this if you are using self-signed certificates. Setting ssl_no_validate to True will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior.
    http_auth: curator_internal:curator_password
    timeout: 30
    master_only: False
  logging:
    loglevel: INFO
    logfile:
    logformat: default
    blacklist: ['elasticsearch', 'urllib3']

Current security privileges for curator_writer role attached to curator_internal user (only applicable for TLS enabled ELK):

curl -k -u elastic:elastic https://<k8s-elasticsearch-service>:9200/_security/role/curator_writer?pretty
{
  "curator_writer" : {
    "cluster" : [
      "manage",
      "cluster:admin/repository/put",
      "cluster:admin/repository/get",
      "cluster:admin/snapshot/status",
      "cluster:admin/snapshot/get",
      "cluster:admin/snapshot/create",
      "cluster:admin/snapshot/restore",
      "cluster:admin/snapshot/delete"
    ],
    "indices" : [
      {
        "names" : [
          "logstash-*"
        ],
        "privileges" : [
          "manage",
          "delete_index",
          "delete",
          "write",
          "read",
          "create"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : {
      "version" : 1
    },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

I've tried a few other combinations as well but what exact Elasticsearch privileges should Curator have? I couldn't find any documentation.

I am seeing these logs in the non-TLS cronjob. Even when it fails, the logs are available. Here are the success logs from the non-TLS cronjob:

kubectl logs -n <some-namespace> elk-non-tls-elasticsearch-curator-2-restore-1584497700-994pg 
2020-03-18 02:37:42,031 INFO      Preparing Action ID: 1, "close"
2020-03-18 02:37:42,038 INFO      Trying Action ID: 1, "close": Close .kibana indices
2020-03-18 02:37:42,083 INFO      Closing 1 selected indices: ['.kibana_1']
2020-03-18 02:37:42,138 INFO      Action ID: 1, "close" completed.
2020-03-18 02:37:42,138 INFO      Preparing Action ID: 2, "restore"
2020-03-18 02:37:42,142 INFO      Trying Action ID: 2, "restore": Restores the .kibana index from the latest snapshot with state SUCCESS
2020-03-18 02:37:46,733 INFO      Restoring indices "['.kibana_1']" from snapshot: kibana-20200318010019
2020-03-18 02:37:46,863 INFO      _recovery returned an empty response. Trying again.
2020-03-18 02:37:56,876 INFO      Provided indices: ['.kibana_1']
2020-03-18 02:37:56,876 INFO      Found indices: ['.kibana_1']
2020-03-18 02:37:56,880 INFO      Found restored index .kibana_1
2020-03-18 02:37:56,880 INFO      All indices appear to have been restored.
2020-03-18 02:37:56,880 INFO      Action ID: 2, "restore" completed.
2020-03-18 02:37:56,880 INFO      Preparing Action ID: 3, "open"
2020-03-18 02:37:56,885 INFO      Trying Action ID: 3, "open": Open .kibana indices
2020-03-18 02:37:56,988 INFO      Opening 1 selected indices: ['.kibana_1']
2020-03-18 02:37:56,993 INFO      Action ID: 3, "open" completed.
2020-03-18 02:37:56,993 INFO      Job completed.

I am seeing the initContainers logs where I configure the s3 repository to restore from (for TLS version):

kubectl logs -n <some-namespace> elk-tls-elasticsearch-curator-restore-cbpq6 -c elasticsearch-s3-repository
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
OK: 7 MiB in 18 packages
waiting for elasticsearch before attempting to configure s3 repository for restore
configuring s3 repository for restore
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   320  100    21  100   299    112   1607 --:--:-- --:--:-- --:--:--  1711
{"acknowledged":true}%

but the main container (that runs the curator action - restore) is not outputting any logs in the TLS version:

kubectl logs -n <some-namespace> elk-tls-elasticsearch-curator-restore-cbpq6 -c elasticsearch-curator

Is this an Elasticsearch role/privilege issue? Or something else?

Looks like the .p12/.pem file was the problem. Tried both types of certs but same result (no output).

When I curled from the es-data node (k8s pod), I got:

[elasticsearch@k8s-elk-elasticsearch-master-2 ~]$ curl --cacert /usr/share/elasticsearch/config/pemCerts/elastic-certificate.pem https://k8s-elk-elasticsearch-master:9200/_cluster/health?pretty
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
[elasticsearch@k8s-elk-elasticsearch-master-2 ~]$ curl --cacert /usr/share/elasticsearch/config/certs/elastic-certificates.p12 https://k8s-elk-elasticsearch-master:9200/_cluster/health?pretty
curl: (77) Problem with the SSL CA cert (path? access rights?)

Which led me to believe the certs were the problem. I resolved this temporarily by doing the following:

# certificate: /etc/es-certs/elastic-certificates.p12 # Absolute path to the root CA
http_auth: curator_internal:curator_password

Reference: https://github.com/elastic/curator/issues/1262