3 times now I've managed to fill up my disks and after recovering the watcher service fails to work reliably. I end up having to restart each data node then it works fine. This isn't just my watches, but the system watches as well. I've tried stopping the service with
followed by a start
it starts erroring right away on the console.
[2018-09-04T22:55:07,503][WARN ][o.e.x.w.e.ExecutionService] [myhost] failed to execute watch [322e52YFQ16I7v2vd9B8kQ_logstash_version_mismatch]
I've not seen any reason for it.
This was just one example.