I don't get it (plugins, data input etc)

[Sjaak01 (Sjaak)]

Hi,

I've just started with ELK last week. So for I managed to get things running and make a few graphs based on syslogs sent by a pfsense box.

However I don't really get ELK. For example I now want to get netflow data into elastic but I got no clue how. I read some guides but there is quite some variation between them and all of them are based on old versions that seem to work different from 5.x as well.

The documentation isn't much use either.

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-netflow.html#_usage_4

For example the output part is very different from what guides tell you. I doubt the way its written in the guide will give you a working filter.

Also for whatever reason there never are any paths mentions. Great that we get some example code but if you don't tell where to put it whats the use?

Same goes for installing plugins. The elastic websites tells me to run bin/logstash-plugin, but doing that simply gives you a no such file or directory error on Ubuntu.

There appears to be no useful documentation at all for beginners to get you up and running, which is very frustrating because it looks like Elastic can do great things but not everybody has 6 weeks of spare time to pick the whole system apart just to get some basic functionality.

Is there any documentation that will get beginners up and running while giving some insight into how ELK does its thing?

[Sjaak01 (Sjaak)]

Okay so somehow netflow data is now coming in but no idea why.

My index is called netflow-%[YYYY.MM.dd} but the index only shows up under logstash-* as netflow first switched and netflow last switched along with @timestamp (should be the one for my syslog files).

Why? Makes no sense at all.

Also the config file is called logstash-netflow.conf but for whatever reason renaming it to netflow.conf breaks things.

Again, no idea why.

I've also put in code => netflow. Seems to work but no idea why because I never installed the codec. Maybe its installed by default? Who knows.

Is there any documentation, resources or whatever that explains at a beginner level how ELK works?

[Sjaak01 (Sjaak)]

Another frustration I have is that there is no information on how to handle multiple hosts sending data to ELK or how to deal with multiple indexes.

E.g. The pfsense tutorial I followed creates three different files, including an output.conf that writes to logstash-%. In the .conf I made for the netflow filter outputs to newflow-% but as newflow appears under logstash-% maybe somehow that is interfering.

But again, the documentation doesn't cover any of that. Or maybe it does but not in an obvious to find place.