Is document_type in filebeat same as type in logstash filter?


(Sri) #1

Hi,

In 5.2.2 (filebeat & logstash), is the document_type setting in filebeat accessible as the type in the logstash filter? I tried this based on the documentation and few other threads, but the logstash fingerprint block does not get executed.

filebeat config section

- input_type: log
  paths:
    - <somepath>/*.out
    - <somepath>/*.log
  fields_under_root: true
  document_type: log

The "fields_under_root" is likely unecessary.

logstash filter sections

filter{
  if "[type]" == "log" {
    fingerprint {
      method => "MD5"
      key => "KEY"
      target => "[@metadata][_id]"
    }
  }

The fingerprint should be executed but it is not.

I've tried a few "random" combinations of if conditions, but no luck so far.

Thanks


(Steffen Siering) #2

The default type should be log. Not sure, but I wonder if it's the condition in logstash does expand fields when being quoted. Does the filter work if you write:

filter{
  if [type] == "log" {
    fingerprint {
      method => "MD5"
      key => "KEY"
      target => "[@metadata][_id]"
    }
  }

(Sri) #3

Thanks @steffens.

This turned out to be a false alarm. The reason I suspected this was because the fingerprinted Id was not reflected in the es document. However, the problem turned out to be later in the pipline : the output filter section didnt set the fingerprinted _id to the "es" document_id field in some conditions.

btw, as you suspected the "" around [type] did not make a difference. And filebeat did set the document_type as "log" by default. I realize some of this stuff changes in 6.0.

A slight digress (maybe new thread?) - in the code I inherited, I see that unix timestamp is prepended to the fingerprinted ID just before the output. Not sure why that is done. It seems to defeat the purpose of the fingerprint to avoid duplicate records inserted in es when they are replayed etc. Any clues?

filter{
  if [type] == "log" {
    mutate {
      add_field => [ "[@metadata][timestamp_copy]","%{@timestamp}" ]
    }
    ruby {
      code => "require 'date';event.set('[@metadata][timestamp_unix]', DateTime.parse(event.get('[@metadata][timestamp_copy]')).strftime('%Q').to_s)"
    }
    mutate {
      replace => [ "[@metadata][_id]","%{[@metadata][timestamp_unix]}_%{[@metadata][_id]}" ]
    }
  }
}

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.