Ahhhh, I thought I was the only one!
We see our logstash come down, with the same log error.
Before the netty exception fires in the log (in fact, for hours before the netty exception appears in the log), we get stacks of these in the log:
[2017-11-24T18:12:59,811][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://<IP of Elasticsearch>:9200/_bulk"}
Then we get the netty error.
How do get guidance on:
a) What logstash monitor url can show us how many open files logstash is holding?
b) What monitor urls can see that logstash is in this hosed state?
c) Aside from the OS config, what do we need to know to truly solve this?