SETUP: I have a kubernetes cluster with 5 nodes, each node has a logstash (replica:1) that is collecting logs from all the pods (via filebeat) on the node, and is writing files on a mounted volume (500 G). I have setup a logrotation pod (can be a side car container in the logstash pod) that sniffs the log files on the same mounted volume and rotates.
QEUSTION: Is it safe for logrotate to copytruncate the log files while logstash is running? If yes then what are the best settings?
In case the above is not safe, I have thought about stopping and starting the pod manually while the gocron - logrotation is in process because filebeat will make sure the logs aren't lost.
But the problem with that is kubetctl picks up the stopped container and restarts the pod. Because it's part of the deployment.
Thanks in advance..