Tcp input writing too much data in log


#1

I started using four tcp inputs simultaneously and I noticed the logs (var/log/logstash/logstash.log) getting bigger and bigger since then. My log is around 28 GB currently, because everything logstash outputs to elasticsearch, it also outputs into this log file (only the tcp inputs though). Is this normal behaviour?


(Mark Walkom) #2

It's not.

Can we see your config?


#3

Which config do you need? The one containing the output or all the ones with tcp inputs?

EDIT: To specify: my only output is to elasticsearch, nonetheless logstash seems to log every incoming message into its own logs... according to this it could be because of errors in the config?


(Magnus Bäck) #4

You're starting Logstash with --verbose or --debug. Don't :smile:


#5

No, I do not use --verbose or --debug.


(Magnus Bäck) #6

Well, either you have a stdout output in your configuration (perhaps in an extra file in /etc/logstash/conf.d?) or it's coming from the logging. What do the logs look like?


#7

I have only one output and it looks like this;

output { elasticsearch { cluster => "name" host => "localhost" index => "%{type}-%{+YYYY.MM.dd}" } }

I just managed to reduce the messages written to the log by replacing two tcp inputs with syslog inputs.

It seems that the remaining log operations are being written because of some encoding error in our filter, which processes Windows Events. I'm going to fix that and update this thread accordingly.

Thank you!


(Mark Walkom) #8

If you can provide a sample of the log it will help.


(system) #9