Need help with filebeat settings to separate logs in logstash for grok parser

[T4iga]

I have an environment with a log-server and a second server runnig an ELK stack. The log-server writes logs to multiple directories on disk. I use filebeat to send those logs to the ELK server.

I wrote a grok parsers to be used to parse the logs from one specific directory on the log-server but not the others. I want to apply the parser to the approriate logs only.

In the logstash config i am planning to use if statements to choose how to parse the logs like the following:

filter {
 if [type] =~ "syslog" {
   grok {
     match => { "message" => " ...
  if [type] =~ "notSyslog" {
   grok {
     match => { "message" => " ...

To set the type field I use a filebeat config with different which adds different entries in the "type" field depending on the folder the log is coming from. It looks as follows:

filebeat.inputs:
- type: log
  enabled: true
  paths:
     - path/*.log
  fields:
    type: myType1
  fields_under_root: true

- type: log
  enabled: true
  paths:
     - path2/*.log
  fields:
    type: myType2
  fields_under_root: true

...

I got filebeat to successfully restart with this config and i can see some traffic between both servers using TCPdump but I can not find any logs with kibana. I create an index "*" which should enable me to find the logs if they were injected at all, but I can't find them.

First Question:

• Is the way I am planning on doing this the correct way?
• Do you see any errors which I need to correct?
• Do i need to use fields_under_root: true or false?

[grumo35]

Hi,

You can indeed separate log sources using filebeat and different log path.

- type: log
    enabled: true
    paths:
        - path: "XXX"
        add_tags:
            tags: "XXX"
        add_fields:

See documentation on add_fields and add_tag

Then in logstash you can use:

if "XXX" in [tags] 
if [field] == "XXX"

one way or another depends on your use case

[Ramicoh]

Hey, to add to grumo35's reply, but in a more general perspective.

As I imagine you're witnessing first hand, writing parsers by yourself is quite a long and laborious effort, and you run the risk of mistakes and wrong interpretation of the data.

Fortunately a lot of the work is already done by others.- e.g. this repository that contains open source parsers which use the Grok processor based on accumulated knowledge, the MITRE ATT&CK scheme and more.

[T4iga]

Thank you for the example. it seems like i had some misundestandings about the correct syntax.
As i understand it, all parts marked with quotes need to be quoted in the config too to denote that it is a string.
How would I add multiple folders under one - path: "XXXX" ?
I suppose I need to insert an array instead of a string if this permitted. I will try using a * first like

- path: "path/a*/*.log"

for replacing 

- path: "path/aa/*.log"
- path: "path/ab/*.log"
- path: "path/ac/*.log"
etc

[T4iga]

Yes, i did witness that. Thank you for the suggestion!
I don't know many features of config yet so I will definitely be able to learn from that repo. Looking at how few fields the parser in that repo actually deals with I still think my own version is much more comprehensive. This might be due to the fact the config in the repo is only for fortinet IPS and mine is for general fortinet logs.
I'll have to see how many bugs mine contains in testing though

[Ramicoh]

Glad that you like the repo! An update of the code is planned soon.

I take it you're writing your own configuration rules - how's that working out? As I pointed out in the last message, this can be a tedious process prone to many errors - see
https://blog.empow.co/loganalysis and https://blog.empow.co/preventing-logstash-misconfiguration.

BTW the repo is open source of course and thus you're most welcome to upload your code to it