I've been trying to troubleshoot this for quite some time and have managed to find out the following:
- filebeat logs everything seemingly fine
- turning on logstash after deleting the indexes i'm logging to SEEMINGLY works fine for a while
until at some point logs literally just stop. - logstash is still logging it's own output fine somehow.
- making my output stdout just prints every log entry to stdout (as expected)
- there is no logstash log file by default in the docker container and the log4j seems to only output errors to console of which there are zero.
Logstash debug logs: Using bundled JDK: /usr/share/logstash/jdk,OpenJDK 64-Bit Server VM warning: O - Pastebin.com
Elastic console output: "stacktrace": ["org.elasticsearch.action.search.SearchPhaseExecutionException: a - Pastebin.com
filebeats config: filebeat.modules:- module: system syslog: enabled: true auth: - Pastebin.com
logstash config: input { beats { client_inactivity_timeout => 1200 port => 5044 } - Pastebin.com
logstash config i changed but STILL doesn't work: input { beats { client_inactivity_timeout => 1200 port => 5044 } - Pastebin.com
All indexes green in health.
All hostnames such as "elasticsearch:9200" in the configs resolve and i can connect via tcp
What's really frustrating is that SOME log lines mystically do seem to occasionally get through but most do not. For instance the logstash console output logs somehow make it with no issue.
EDIT: HEre's my filebeeat log which has plenty of errors but they seem unsolvable and I think they do that even after I delete indexes and logging seems to be working again:
filebeat.log (not pastebin because pastebin thinks it's questionable content somehow)