Hello,
I have a cron job set up in my crontab which is set to fire a snapshot every first minute of an hour. When I see the cron logs I can see that it triggered the task but I do not see the snapshots in Kibana client. I curl'ed the ElasticSearch instance directly and do not see the snapshots there either. When I try to run the cron script manually from one of the nodes it works fine and creates a snapshot but surprisingly the crontab job does not do so.
Here is my cron job setting
< 1 * * * * root curl -v -k -s -X PUT https://redacteduser:reactedpassword@localhost:9201/_snapshot/s3_backup_repository/<snapshot-{now%2Fh{yyyy.mm.dd-hh}}>?wait_for_completion=true >> /opt/elasticsearch/system/system.log 2>&1 />
This cron job is set on all the three EC2 instances in our ES cluster.
Here is the row in cron log in /var/log/ which shows the script was triggered per the cron schedule. (I see this in all the 3 EC2 instances in ES Cluster all with the same timestamp).
< Sep 28 18:01:01 ip-10-7-22-136 CROND[17730]: (root) CMD (curl -v -k -s -X PUT https://redacteduser:redactedpassword@localhost:9201/_snapshot/s3_backup_repository/) />
The system.log file in all the 3 instances on path /opt/elasticsearch/system do not have any trace of the snapshot triggered by the cron tasks.
Any idea why this could be happening? Is it because all the 3 instances in the cluster try to trigger the snapshot all the same time?
Thanks,
Rahul