Processing Custom Log Input from Filebeat in Logstash

[luke14]

Hi

I have setup Filebeat on a server which I would like it to pass a custom application log to my logstash server.

On my logstash server I currently have a beats_input.conf which is listening for all my beats inputs. And a beats_output.conf which sends the data to elasticsearch. All this functionality is working well.

If I want to add in my custom log file, on my logstash server do I need a beats_postprocess.conf with my GROK filter in it. Or can I add my filter to the beats_input.conf?

What is the best practice here? All my other beats inputs are from Winlogbeat and are being processed by the beats_input.conf and the beats_output.conf file on my logstash server.

Any guidance would be appreciated.

Thanks

[pup_seba]

Hi mate,

I'm just starting with elastic stack, but for what i can see in the courses and books, you could have it all in one file if you wanted to, although most of the times they suggest having it all separated in different files.

As for my understanding, those files are read in alphabetical order, thus, you'll find that in lots of places they do/show something like.
10_input_beats.conf
20_filter_something.conf
21_filter_somethingelse.conf
30_output_elasticsearch.conf

For instance, you could see something like this, really easily if you get the elk docker from docker hub and see how they pre-configure those files in there.

Regards,

[luke14]

Hi,

Thanks for the reply.

So If I place my GROK filter in a separate file, I assume only the beats input that MATCH that particular GROK filter will be processed by that file and everything else will simply fall through and simply just be output to elasticsearch as it is now?

Thanks

[pup_seba]

Hi,

In my case, it is working exactly like that, and also, that's what I understood from the things I've studied...so yes, if you place your GROK filter in a separate file, only the "inputs" (you could use other inputs beside beats) that match your "filters" (you can use other filters beside grok) will be processed by that file or files.

Just remember that things are applied alphabetically, so inputs before filters and filters before outputs (thus using numbers in front of the filenames).

For what I can understand, the reason for using different files is more a thing of ease of reading and ease for changing things, than a functionality thing.