Elasticsearch version: 6.8.2
Elasticsearch Curator version: 5.7.6
My action files were working fine before I enabled TLS.
So in order to make sure, I spun up 2 instances of ELK (1 with and 1 without TLS) using helm and Kubernetes. Everything is identical except the extra secret volume mount and some security settings.
config.yml:
config.yml: |-
---
client:
hosts:
- 127.0.0.1
port: 9200
# url_prefix:
use_ssl: True # If set to True, curator will connect with HTTPS instead of HTTP
certificate: /etc/es-certs/elastic-certificates.p12 # Absolute path to the root CA
# client_cert and client_key are used for Client Certificate Authentication
# client_cert: /etc/elasticsearch/config/client_certificate.pem # Absolute path to the client TLS certificate that is sent with each request.
# client_key: /etc/elasticsearch/config/private_key.pem # Absolute path to the private key of the client certificate.
ssl_no_validate: True # If set to True curator will not validate the certificate it receives from Elasticsearch. Enable this if you are using self-signed certificates. Setting ssl_no_validate to True will likely result in a warning message that your SSL certificates are not trusted. This is expected behavior.
http_auth: curator_internal:curator_password
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']
Current security privileges for curator_writer role attached to curator_internal user (only applicable for TLS enabled ELK):
curl -k -u elastic:elastic https://<k8s-elasticsearch-service>:9200/_security/role/curator_writer?pretty
{
"curator_writer" : {
"cluster" : [
"manage",
"cluster:admin/repository/put",
"cluster:admin/repository/get",
"cluster:admin/snapshot/status",
"cluster:admin/snapshot/get",
"cluster:admin/snapshot/create",
"cluster:admin/snapshot/restore",
"cluster:admin/snapshot/delete"
],
"indices" : [
{
"names" : [
"logstash-*"
],
"privileges" : [
"manage",
"delete_index",
"delete",
"write",
"read",
"create"
],
"allow_restricted_indices" : false
}
],
"applications" : [ ],
"run_as" : [ ],
"metadata" : {
"version" : 1
},
"transient_metadata" : {
"enabled" : true
}
}
}
I've tried a few other combinations as well but what exact Elasticsearch privileges should Curator have? I couldn't find any documentation.
I am seeing these logs in the non-TLS cronjob. Even when it fails, the logs are available. Here are the success logs from the non-TLS cronjob:
kubectl logs -n <some-namespace> elk-non-tls-elasticsearch-curator-2-restore-1584497700-994pg
2020-03-18 02:37:42,031 INFO Preparing Action ID: 1, "close"
2020-03-18 02:37:42,038 INFO Trying Action ID: 1, "close": Close .kibana indices
2020-03-18 02:37:42,083 INFO Closing 1 selected indices: ['.kibana_1']
2020-03-18 02:37:42,138 INFO Action ID: 1, "close" completed.
2020-03-18 02:37:42,138 INFO Preparing Action ID: 2, "restore"
2020-03-18 02:37:42,142 INFO Trying Action ID: 2, "restore": Restores the .kibana index from the latest snapshot with state SUCCESS
2020-03-18 02:37:46,733 INFO Restoring indices "['.kibana_1']" from snapshot: kibana-20200318010019
2020-03-18 02:37:46,863 INFO _recovery returned an empty response. Trying again.
2020-03-18 02:37:56,876 INFO Provided indices: ['.kibana_1']
2020-03-18 02:37:56,876 INFO Found indices: ['.kibana_1']
2020-03-18 02:37:56,880 INFO Found restored index .kibana_1
2020-03-18 02:37:56,880 INFO All indices appear to have been restored.
2020-03-18 02:37:56,880 INFO Action ID: 2, "restore" completed.
2020-03-18 02:37:56,880 INFO Preparing Action ID: 3, "open"
2020-03-18 02:37:56,885 INFO Trying Action ID: 3, "open": Open .kibana indices
2020-03-18 02:37:56,988 INFO Opening 1 selected indices: ['.kibana_1']
2020-03-18 02:37:56,993 INFO Action ID: 3, "open" completed.
2020-03-18 02:37:56,993 INFO Job completed.
I am seeing the initContainers logs where I configure the s3 repository to restore from (for TLS version):
kubectl logs -n <some-namespace> elk-tls-elasticsearch-curator-restore-cbpq6 -c elasticsearch-s3-repository
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
OK: 7 MiB in 18 packages
waiting for elasticsearch before attempting to configure s3 repository for restore
configuring s3 repository for restore
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 320 100 21 100 299 112 1607 --:--:-- --:--:-- --:--:-- 1711
{"acknowledged":true}%
but the main container (that runs the curator action - restore) is not outputting any logs in the TLS version:
kubectl logs -n <some-namespace> elk-tls-elasticsearch-curator-restore-cbpq6 -c elasticsearch-curator
Is this an Elasticsearch role/privilege issue? Or something else?