Hi,
I am implementing an ELK Stack and have got the server up and running with all the services and applications. Everything is working fine as I can see all the syslogs from routers and switches in Kibana.
I have a question. We also need to capture NETFlow, but I am wondering whether it will get mixed up into the syslogs.
Do I need to create a seperate index? Right now I have two, "logstash_syslogs*" and "logstash_netflow*". The default one on Kibana is "logstash_syslogs*" and can see them all coming in.
Anyone who can assist me in visualizing NETFlow as well? Do I need to switch between indexes to do so and will I have to create a filter for my NETFlow config file?
Here are my configurations:
logstash_syslog
http://pastebin.com/uyyY5UFz
logstash_netflow
http://pastebin.com/B6Y6fiHm
I don't have any experience with NETFlow or Syslog, but I can speak to your use of Kibana a bit. Perhaps that will be helpful...
If you're indexing this information in 2 different indexes, then yes, you will have to swap around to use them in Kibana. You can create different visualizations using the different indices and put both of them on the Dashboard. If that's good enough for you, then your setup is fine. However, if you want to visualize Syslog and NETFlow data in the same visualization, this won't be possible.
You could add a 3rd index pattern for logstash_*, which would match both indices, and would then allow the data to be combined in a single visualization. Alternatively, you could potentially index these documents in the same index, but with different types (you can create field mappings by index and type), and use filters on _type to select the data you want to work with. This may end up being a little more cumbersome though.
With both options, all the fields from both indices would show up in the aggregation options, and if they don't overlap, the results won't be useful anyway.
I suspect your use of different index patterns in Kibana (and different indices in Elasticsearch) is probably what you want.
1 Like
Alright, that makes sense. I'll try out creating visualizations on both indexes and create a general dashboard for a good overview of the data.
Although I would prefer working with a single index, but I thought having two different types of data in one index could conflict. Is this not the case? Will fields from both types of data be available to use and create visualizations from?
That's true, and what I meant by the data overlapping. Basically, if you don't have the same fields, with the same types on those fields, across both of the data types, it won't work the way you probably want.