Filebeat to add extra fields for logstash 7.17, it worked previously but not anymore?

Elastic Stack Beats
1

I have my dmarc interpreter running here, and noticed that it didn't produce any data to the kibana.
It looks like the config is ignored with the 7.17, and back when it was 6.x it worked.
Can you tell me what I've done wrong with the config, I tried searching for it, but without any luck.
The filebeat.yml:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

#filebeat.registry_file:  filebeat_registry.json

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /dmarclogs/*.json
  json.keys_under_root: true
  json.add_error_key: true
  fields_under_root: true
  fields:
    source_type: json-logs
    logtype: dmarc

output.logstash:
      hosts: ["logstash:5000"]

In logstash, it should then be captured by this config:

input {
        beats {
                port => 5000
                ecs_compatibility => disabled
        }
        syslog {
        }
}
filter {
  if "dmarc" in [fields][logtype] {
    if [source_type] == "json-logs" {
      json {
        source => "."
        tag_on_failure => ["_jsonparsefailure"]
      }
    }
    geoip {
      source => "source_ip"
    }
    if "_geoip_lookup_failure" not in [tags] {
      mutate {
        add_field => {
          "[geoip][location][coordinates]" => "%{[geoip][location][lat]}, %{[geoip][location][lon]}"
        }
#        remove_field => ["host"]
      }
    }
     date {
       match => ["date_end" , "yyyy-MM-dd'T'HH:mm:ss"]
       timezone => "UTC"
       target => "@timestamp"
     }
#    fingerprint {
#      id => "Duplicate Protection"
#      source => "message"
#      target => "[@metadata][fingerprint]"
#      method => "MURMUR3"
#    }
    mutate {
      add_field => {
        "ingest_time" => "%{+YYYY.MM.dd HH:mm:ss}"
      }
    }
  }
}
output {
  elasticsearch {
    hosts => "https://elasticsearch:9200"
      index => "%{[fields][logtype]}-%{[@metadata][version]}-%{+YYYY.MM}"
      document_type => "%{[@metadata][type]}"
      cacert => "/usr/share/logstash/config/certs/ca/ca.crt"
      user => xxx
      password => xxx
  }
}

But in kibana the index is shown as:
%{[fields][logtype]}-7.17.4-2023.07

So it looks like the logtype is empty, right?

2

It looks like that if I remove the

fields_under_root: true

Then it works again?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.