Add multiple types in filebeat for logstash output

Elastic Stack Logstash
1

Is there a way I can have multiple types with filebeat and logstash? I have multiple app logs I want to send to different indices. BUT they will be processed the same way

I want to do something like this:

filebeat.prospectors:
- type: iis,app01
  paths:
    - /c:/intepub/app01

- type: iis,app02
  paths:
    - /c:/intepub/app02

  fields_under_root: true

And then in logstash something like this:

input {
  tcp {
    type => "iis"
    port => "9999"
  }
}

filter {
  if [type] == "iis" {
  ...process iis logs....
  }
}

output {
  if [type] == "app01" {
    elasticsearch {
      hosts => "${ES_CLUSTER}"
      index => "logstash-APP01-%{+YYYY-MM-dd}"
    }
  }
  if [type] == "app02" {
    elasticsearch {
      hosts => "${ES_CLUSTER}"
      index => "logstash-APP02-%{+YYYY-MM-dd}"
    }
  }
}

Is this possible or I'm I going about this the wrong way?

2

Yes, you can do this. You can add tags (or fields) in the prospector configuration and make everything in the logstash configuration conditional upon what tags the event has (or the value of some field, if you want to add a field in filebeat).

3

But then I would have to modify the grok config right?

So in filebeat I could do this:

filebeat.prospectors:
- type: iis
  paths:
    - /c:/intepub/app01
fields:
  app_id: app01

- type: iis
  paths:
    - /c:/intepub/app02
fields:
  app_id: app02

  fields_under_root: true

But then does that field make my log line different?

Right now I have a big grok line:

grok {
  match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:serviceName} ...etc }
}

These filebeat fields don't mess up my grok do they? Can I just referent them in the output somehow like this:

output {
  if [app_id] == "app01" {
    elasticsearch {
      hosts => "${ES_CLUSTER}"
      index => "logstash-APP01-%{+YYYY-MM-dd}"
    }
  }

Actually is there a way I could reference the field like this, this would be even better:

output {
    elasticsearch {
      hosts => "${ES_CLUSTER}"
      index => "logstash-{app_id}-%{+YYYY-MM-dd}"
    }
}
4

No. Filebeat puts the line from the file in the event field message. It then adds the additional fields or tags that you configured. They are attached as fields or tags, they do not modify the message field.

I believe the "even better" substitution would work if you use a %{} reference.

index => "logstash-%{app_id}-%{+YYYY-MM-dd}"
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.