Trying to use GROK to filter data to different index

[snovak]

We have been using GROK to filter and parse message date from the Unbound DNS server files coming in via FileBeat for some time. Now we are starting to send some other data with FileBeat and want to separate out the Unbound data to its own index. We have tried to do that with the following GROK filter and output IF statement but it doesn't seem to be working. None of the Unbound data is showing up in the Filebeat indices anymore, but it isn't creating the Unbound indices either.

input {
  beats {
    port => 5045
    host => "IP"
  }
}

filter {
  grok {
    id => "unbound"
    match => { "message" => "%{GREEDYDATA:datestamp} %{USER}\[%{NUMBER:unbound.process_id}:%{NUMBER:unbound.instance_id}] %{LOGLEVEL}: %{IP:unbound.query.client_ip} %{GREEDYDATA:unbound.query}\. %{WORD:unbound.query.record_type} IN %{WORD:unbound.query.message_flags} %{NUMBER:unbound.query.duration} %{NUMBER:unbound.query.cached} %{NUMBER:unbound.query.cache_id}" }
  }
}


output {
  if ${id} == "unbound" {
    elasticsearch {
        hosts => "IP:9200"
        index => "unbound-%{+YYYY.MM.dd}"
    }
  } else {
    id => "filebeat"
    elasticsearch {
        hosts => "IP:9200"
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

[pjanzen (Paul Janzen)]

Try

if [id] == "unbound"

The ${} is more of a shell notation

[snovak]

Hmm, sadly that didn't seem to fix it.

[snovak]

Figured out how to make it work. I couldn't get it to work with the ID field, so I did the following:

1. I changed the filebeat to add a tag for the unbound logs

   - type: log
     enabled: true
     paths:
       - /var/log/unbound.log
     input_type: log
     scan_frequency: 2
     tags: ["unbound", "DNS"]
     exclude_files: ['.gz$']
   

2. I changed the pipeline configuration to look for the tag:

   output {
   if "unbound" in [tags] {
   elasticsearch {
   hosts => "IP:9200"
   index => "unbound-%{+YYYY.MM.dd}"
   }
   } else {
   elasticsearch {
   hosts => "IP:9200"
   index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
   }
   }
   }