Logstash does not process logs in time and so the application sending log message backs up

Elastic Stack Logstash
1

Hi:
I have a java application that sends an warning message whenever it hits an error during its analyzing a document.
I am processing several millions of these documents in a day, and the application is generating several 100's of thousands of error messages. The application is also generating other log messages.

All of these log messages are being sent via a logback -logstash appender to logstash listening on a TCP socket.
I have logstash configured to have a tcp input.

I am noticing that after a while, logstash falls behind in the processing of messages and the application starts backing up. This causes the application to go into a "wait" state.

there are a couple of problems here:

  1. Application is stalling up on not being able to sending log messages
  2. logstash is backing up trying to process the messages.

To address problem 2, are there configuration settings on the TCP input that would allow for multiple threads (like the UDP input) ?

to get around Problem 1, if I restart the application, logs start going through once again.

I am running Logstash 2.2.2

any thoughts on this

Thanks
Ramdev

2

If you really have to send the data directly to Logstash via TCP, use an intermediate broker as a buffer.

I'd use a local file as the buffer to guarantee (barring disk space shortage) that the logs can always be consistently offloaded from the application.