How to configure conditional input-output from Filebeats

[74db36a597f21b891b3f (PaWeLL)]

Hi there. I'm a newbie in ELK stack. I am trying to configure logstasth to gather data from filebeat and put it in different indices depending from sources' filenames.

Filebeats config:

filebeat.inputs:
- type: log
enabled: true
paths:
   - D:\Logs\UIS\CMS\*
fields:
log_type: cmslog
fields_under_root: true
- type: log
enabled: true
paths:
   - D:\Logs\UIS\MonitoringService\*
fields:
log_type: monlog
fields_under_root: true

Logstash config:

input {
  beats {
    port => 5044
  }
}
filter{
 if [fields.log_type] == "cmslog" {
  grok{
  match=>{ "message" => "%{DATE_EU:date}\s*%{TIME:time}\s*\[%{DATA:thread}\]\s*\[%{DATA:username}\]\s*\[%{LOGLEVEL:loglevel}\]\s*\[%{DATA:logger}\]\s*\[%{DATA:someguid}\]\s*%{NO$
  }
 }
}
output {
  if [fields.log_type=="cmslog"]{
  elasticsearch {
    hosts => ["host:9200"]
    sniffing => true
    index => "cmslogs-%{+YYYY.MM.dd}"
    manage_template => false
   }
  }
}

With these configs Elasticsearch get no data.
How to write conditions correct or debug why it's not working?

[magnusbaeck (Magnus Bäck)]

You're using the wrong syntax for nested fields, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

[74db36a597f21b891b3f (PaWeLL)]

Hello, magnusbaeck.

Thank for the reference. Should I use [log_type=="cmslog"] instead of [fields.log_type=="cmslog"].
This config is not working too. Is my custom field on top-level. And if not, which field is on top?

[magnusbaeck (Magnus Bäck)]

Since you have fields_under_root: true you should use [log_type] == "cmslog". But there's no need to speculate; skip the conditionals and inspect what your events actually look like, then adjust your configuration to suit reality.

[74db36a597f21b891b3f (PaWeLL)]

Without filters my data just comes through logstash without problem.

[magnusbaeck (Magnus Bäck)]

Yes, but what does an example event look like?

[74db36a597f21b891b3f (PaWeLL)]

All data is being dropped to one index. Even grok filter is working.
Problem is to conditionally allocate different log by indices.

[magnusbaeck (Magnus Bäck)]

Yes, I understand what the problem is. If you want help to resolve this please answer my questions.

[74db36a597f21b891b3f (PaWeLL)]

Hello. Sorry for misunderstanding.
Here is the sample event.

[magnusbaeck (Magnus Bäck)]

You have misindented your field_under_root: true line in the Filebeat configuration. It should be on the same level as the fields: lines. If you fix that fields.log_type will become plain log_type and your Logstash configuration should read if [log_type] = "..." {.

[74db36a597f21b891b3f (PaWeLL)]

Thank you very much. It helps.