Logstash - With Filebeats and Netflow

Elastic Stack Logstash
1

Hi all,

Been working with the ELK stack and loving it. A bit of a learning curve but finally got all working with dashboards etc.

I have file beat reporting back from a FW and it is working really well to display my dashboards. I wanted to get net flow working as well.

How I tried to get it working:

  • I used the cmd line option as per the documentation and set my UDP port as specified in the manual.
    What I noticed is that it started a bootstrap of LogStash with Net Flow running instead of intergrating it into the current running LogStash. All Dashboards & Vis were created and the pipeline was created in ES.
    When I connected to the running LogStash process to see what was happening I didn't see any "netflow type of activity" - all other activity was as per normal.

  • I then tried option 2 - specifying the module and the var's in the logstash.config. Restarted the entire ELK stack. In this case

  1. File Beat stopped sending or ELK was not receiving any info from the file beat agent on the FW.
  2. No info was being passed to the ES pipeline either.
    I checked the LogStash log files and saw that it started up and the UDP port was created as specified but still no File Beats.

Then when I remove (comment out the Netflow) module restart the ELK stack.

LogStash starts up as per normal (in the logs). But I have to go to ES close the pipeline and then reopen it. After 30s - 1min the items start flowing in again.

I guess my question(s) are:

  1. Can you have File Beat and Net Flow "talking" to the same LogStash instance (my lamen view is there seems to be some clash here)

  2. Why would I have to close and then reopen ES pipeline for logstash after disabling a module to get the flow of info in?

Thank you for the time and looking forward to the response

2

You may want to consider using ElastiFlow. The Logstash Netflow Module was based on ElastiFlow 1.0.0, which is really old now.

3

Thank you sir will try that out! Thanks once again

4

Just an additional question will this work with flows from pfsense firewall?

5

I am not aware of any issue. If you encounter one, please open an issue on the GitHub repository.

6

I have tried to get ElastiFlow to run on my SuperMicro server for the weekend. Still can't figure out what is going on.

Says starting pipeline - no error nothing. I can see logstash load the config, I can see logstash starting with the UDP listener. I can see it saying start elastiflow pipeline - no errors after that either.

Had many different ones had the java heap trying to store the cipher key in tmp which is noexec and one doesnt change that. Fixed that issue etc etc etc. Now if I could just solve the pipeline issue. Dashboards load the index pattern wont but I assume thats because the pipeline isnt created.

Machine has Xeon and 128gig ram and heap sizes for Elastic Kibana and logstash are all 24gigs each as thats all that machine does.

Just not sure were to go with this anymore.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.