Here is our ELK setup and issue we are facing with our use case.
We have multiple filebeat agents that ship logs to logstash where it groks and sends to elastic search servers.
We have close to 150 VMs in our environment and all servers have shared file system (NAS- /nas/apps).
We installed one Filebeat 5.4 software instance in /nas/apps/filebeat and started individual filebeat agents using below command on each of 150 VMs.
nohup /nas/apps/filebeat -c /nas/apps/filebeat/config/${hostname}.filebeat.yml > $logDir/filebeat.log 2>&1 &
Each VM has its own config file ( ${hostname}.filebeat.yml ) but all share same instance of filebeat. We are facing issues with registry file with this approach.
ERR Error when encoding the states: write /nas/apps/filebeat/data/registry.new: stale NFS file handle
2017-06-05T06:18:22-04:00 ERR Writing of registry returned error: write /nas/apps/filebeat/data/registry.new: stale NFS file handle. Continuing...
2017-06-05T06:18:30-04:00 ERR Rotate error: rename /nas/apps/filebeat/data/registry.new /nas/apps/filebeat/data/registry: no such file or directory
2017-06-05T06:18:30-04:00 ERR Writing of registry returned error: rename /nas/apps/filebeat/data/registry.new /nas/apps/filebeat/data/registry: no such file or directory. Continuing...
I read in some of forums that each filebeat agent should have its own registry file. Is there a way I can start filebeat instance by specifying its own registry file from single software installation? Just dont want to install 150 filebeat agents as upgrading and maintaining will be difficult for us. Any help with this is really appreciated.