I have a java application that sends an warning message whenever it hits an error during its analyzing a document.
I am processing several millions of these documents in a day, and the application is generating several 100's of thousands of error messages. The application is also generating other log messages.
All of these log messages are being sent via a logback -logstash appender to logstash listening on a TCP socket.
I have logstash configured to have a tcp input.
I am noticing that after a while, logstash falls behind in the processing of messages and the application starts backing up. This causes the application to go into a "wait" state.
there are a couple of problems here:
- Application is stalling up on not being able to sending log messages
- logstash is backing up trying to process the messages.
To address problem 2, are there configuration settings on the TCP input that would allow for multiple threads (like the UDP input) ?
to get around Problem 1, if I restart the application, logs start going through once again.
I am running Logstash 2.2.2
any thoughts on this