Problem of collecting different types of logs( Netflow + Syslog+ Beats) at the same time with logstash version 6.2

Elastic Stack Logstash
1

hello,
I installed for the first time elastic stack on centos 7, I want to collect different types of logs (firewalls logs through syslog, and windows logs through winlogbeat) and netflow v9.
I managed to collect and view windows logs through Winlogbeat, but when I add the configuration file "netflow.conf" in /etc/logstash/conf.d, logstash stops.
Apparently it is not possible to collect them all at once. So I deleted the conf file "beats.conf" and I left only "netflow.conf", I activated the netflow module with the command:
#bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 -M netflow.var.elasticsearch.hosts="localhost:9200"
I was able to collect it and visualize it on kibana . but, the netflow dashboard is not created on kibana.

To summarize, I would like to know please:

  • How can I collect all these types of logs at once (must I change the path.data for each configuration file (netflow / beats / syslog))?
  • How to create the dashboard for the netflow on Kibana?
2

Yes you can, read the pipeline config section. Before pipelines, I had a lot of logic in my logstash config to select input, process and store it. With pipelines, each input-filter-output is a individual file. Some of my beats configs do a lot of filtering, it can use more cpu than simple filters.

3

I've not used the netflow module, but this says there are dashboards provided: https://www.elastic.co/guide/en/logstash/current/netflow-module.html

4

You can certainly collect all those things (we do). You will have to control your pipeline yourself though, and not just drop the modules in the config dir.

If you want consistent field names, you'll want to tweak your own filters. The modules can make good starting points.

5

Thank you for your answers.

Indeed, for the netflow, I opted for Elastiflow, I installed it and configured it (adding the pipeline id and the config path in pipeline.yml). I can visualize the netflow now on kibana, I could also import the dashboard. Except that I don't manage to have all the graphs (for Geoip for example yet I have the geographical coordinates). Moreover, I just have daily indexes like netflow-YYYY-MM-DD that I add every day and not the ability to select the "netflow- *" index. While for Winlogbeat, I select it on "discover" without adding the index each time

For windows logs, I just added once again the pipeline ID for beats and its config path (different from elastiflow) and it worked, I was able to collect both types of logs (netflow + windows logs). But, for the Winlogbeat Dashboard I have the following error: **saved "field" parameter is now invalid. I found the field, indeed there is confusion it has two different format " text" and "date".

For firewalls logs that I want to recover from another SIEM through the syslog protocol port 514, I did the same thing (add in the pipeline.yml). But, it did not work and I have as error "Syslog listener died {: protocol =>: udp,: adress =>" 0.0.0.0:514 ",: exception => # <SocketError: bind: name or service not known, ...

To summarize, I still have a problem with the collection of firewalls logs with syslog, the dashboad of Winlogbeat and the Geoip graphs of netflow .... thank you in advance for help.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.