Logstash works for a few minutes then stops

Elastic Stack Logstash
1

This is a new instance for testing purposes in a personal lab.

After setting up logstash and adding configuration I start the service. Logstash proceeds to run for a minute or two and just completely stops either receiving logs or sending them to elastic.

I'll stop/start the service and the same behavior happens.

I checked /var/log/logstash for the logs to find no errors/warnings.

Currently sending winlogbeats and netflows to the logstash instance.

Running all the latest components of ELK on a centos 7 server.

Any help idea are much appreciated!

2

Do you have debug logging enabled?

3

turned on debug and took a look at the log file.... there's a lot more info than i know what to do with.

any idea what i'm looking for to possibly figure out what's going on?

4

So debug logs seem to show nothing, the logs seem to be the same as it was before it stops gathering new events.

I could have missed something however, as the volume of logging produced by enabling debug is enormous.

5

Yes it is, lol. There should be a FATAL log event in there though.

6

No instance of Fatal log events in the debug log file...

Tried pulling logstash out of the instance and giving it it's own VM, but I ran into the same issue. Logstash ships logs for a couple minutes then dies and stops all together.

Debug log shows that it seems to still be working business as usual despite elastic not receiving logs/netflows.

I'm stumped :frowning:

7

Interesting...I've run into Logstash performing the way you say but usually there is a FATAL event generated in the logs. Do you have any monitoring configured with X-Pack? What do you have the JVM memory heap set to?

8

yea no FATAL event.

The heap was set to 256m/1g, but wasn't seeing any issues with the cpu usage. I bumped it to 2g/2g to see if that helped, but still got the same issue.

Bumped the heap size for elastic as well to see if it that helped any, still same issue as before :frowning:

Unfortunately haven't installed x-pack, i'll need to look into setting that up.

9

So i figured out what the issues was!

Turns out because netflows and winlogbeats were going over the same pipe with no filtering setup the events were getting mixed up. As soon as i created two different pipes for the different logs, events were ingesting just fine.

Would never have figured that out if I had never installed the x-pack.

Thanks!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.