How to configure conditional input-output from Filebeats

Hi there. I'm a newbie in ELK stack. I am trying to configure logstasth to gather data from filebeat and put it in different indices depending from sources' filenames.

Filebeats config:

filebeat.inputs:
- type: log
enabled: true
paths:
   - D:\Logs\UIS\CMS\*
fields:
log_type: cmslog
fields_under_root: true
- type: log
enabled: true
paths:
   - D:\Logs\UIS\MonitoringService\*
fields:
log_type: monlog
fields_under_root: true

Logstash config:

input {
  beats {
    port => 5044
  }
}
filter{
 if [fields.log_type] == "cmslog" {
  grok{
  match=>{ "message" => "%{DATE_EU:date}\s*%{TIME:time}\s*\[%{DATA:thread}\]\s*\[%{DATA:username}\]\s*\[%{LOGLEVEL:loglevel}\]\s*\[%{DATA:logger}\]\s*\[%{DATA:someguid}\]\s*%{NO$
  }
 }
}
output {
  if [fields.log_type=="cmslog"]{
  elasticsearch {
    hosts => ["host:9200"]
    sniffing => true
    index => "cmslogs-%{+YYYY.MM.dd}"
    manage_template => false
   }
  }
}

With these configs Elasticsearch get no data.
How to write conditions correct or debug why it's not working?

You're using the wrong syntax for nested fields, see https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references.

Hello, magnusbaeck.

Thank for the reference. Should I use [log_type=="cmslog"] instead of [fields.log_type=="cmslog"].
This config is not working too. Is my custom field on top-level. And if not, which field is on top?

Since you have fields_under_root: true you should use [log_type] == "cmslog". But there's no need to speculate; skip the conditionals and inspect what your events actually look like, then adjust your configuration to suit reality.

Without filters my data just comes through logstash without problem.

Yes, but what does an example event look like?

All data is being dropped to one index. Even grok filter is working.
Problem is to conditionally allocate different log by indices.

Yes, I understand what the problem is. If you want help to resolve this please answer my questions.

Hello. Sorry for misunderstanding.
Here is the sample event.

You have misindented your field_under_root: true line in the Filebeat configuration. It should be on the same level as the fields: lines. If you fix that fields.log_type will become plain log_type and your Logstash configuration should read if [log_type] = "..." {.

1 Like

Thank you very much. It helps.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.