Logstash happily plows along for an indeterminate amount of time (sometimes a couple days, sometimes a couple hours) and then nothing else is passed through to ES. At this point, the system is nearly idle (the only way I've yet to "monitor" for this sad symptom is to look at top/htop for an abnormally idle system and to query kibana/ES to find there is no new data). ES reports LS as being connected at this point. LS will not HUP or cleanly restart. An ungraceful kill (-9) and restart of LS is required. I've turned up LS logging to --verbose, but am still finding nothing telling in the logs. I'm glad to provide an strace of LS while it is "out to lunch" if that would be useful. Let me/us know what we can do to help further diagnose.
I've gone through my config and wrapped all my conditionals with an extra conditional (to verify the field exists before querying against its value) so as to avoid this (https://goo.gl/XDd4kH) possible problem...
CentOS Linux release 7.1.1503 (Core)
java version "1.7.0_79"
OpenJDK Runtime Environment (rhel-184.108.40.206.el7_1-x86_64 u79-b14)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)