Filebeat adds a host.name field to JSON messages even when already set

[fgabolde (Fabrice Gabolde)]

I'm using an UDP input to gather logs from Java applications with Filebeat. The Java apps send JSON objects over UDP through https://github.com/logstash/logstash-logback-encoder.

I had some issues since it seems that the UDP input wraps the messages in a JSON object that I had to decode explicitly... I feel like it was more work than necessary, but I got it to work after I found Filebeat 6.3.1 UDP + Json. The app is successfully sending logs to Filebeat, Filebeat is decoding them from JSON-in-JSON, the Elasticsearch output works fine,

Now I'm reading through the Elastic Common Schema to make my logged JSON objects as compliant as possible and it specifies pretty clearly (https://www.elastic.co/guide/en/ecs/1.0/ecs-host.html) that host.name should contain the name of the host from which the logs originated; meaning, the app's host. Unfortunately, the host.name field is already populated by Filebeat somehow, set to its own hostname. As I understand it that data should be somewhere in the agent.* fields instead (and indeed agent.hostname has the correct value).

I replaced the Filebeat container with https://github.com/mendhak/docker-udp-listener just to check that the app was sending host.name with the right value, and it is.

This is my Filebeat config:

filebeat.inputs:
- type: udp
  max_message_size: 10KiB
  host: "filebeat:8080"
  enabled: true
  encoding: utf-8
  processors:
  - decode_json_fields:
      # see e.g.
      # https://discuss.elastic.co/t/filebeat-6-3-1-udp-json/140624/2
      fields: ["message"]
      target: ""
      overwrite_keys: true

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'
  enabled: false

output.console:
  pretty: true
  enabled: true

logging.level: debug
logging.to_stderr: true

And this is the console output:

filebeat_1_d541bdaf93da  | {
filebeat_1_d541bdaf93da  |   "@timestamp": "2019-07-16T15:33:37.744Z",
filebeat_1_d541bdaf93da  |   "@metadata": {
filebeat_1_d541bdaf93da  |     "beat": "filebeat",
filebeat_1_d541bdaf93da  |     "type": "_doc",
filebeat_1_d541bdaf93da  |     "version": "7.2.0",
filebeat_1_d541bdaf93da  |     "truncated": false
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "message": "<snip>",
filebeat_1_d541bdaf93da  |   "log": {
filebeat_1_d541bdaf93da  |     "level": "ERROR",
filebeat_1_d541bdaf93da  |     "package": <snip>",
filebeat_1_d541bdaf93da  |     "thread": "<snip>"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "input": {
filebeat_1_d541bdaf93da  |     "type": "udp"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "container": {
filebeat_1_d541bdaf93da  |     "short_id": "8fd8a7418838"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "agent": {
filebeat_1_d541bdaf93da  |     "id": "10bf9f27-bc3b-4804-a523-7b110bfd9ef5",
filebeat_1_d541bdaf93da  |     "version": "7.2.0",
filebeat_1_d541bdaf93da  |     "type": "filebeat",
filebeat_1_d541bdaf93da  |     "ephemeral_id": "e64c1613-100a-44ee-a688-db744d22fca4",
filebeat_1_d541bdaf93da  |     "hostname": "fe62ee45f705"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "ecs": {
filebeat_1_d541bdaf93da  |     "version": "1.0.0"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "service": {
filebeat_1_d541bdaf93da  |     "name": "logback-elasticsearch-appender-demo",
filebeat_1_d541bdaf93da  |     "version": "1.0.0-SNAPSHOT"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "labels": {
filebeat_1_d541bdaf93da  |     "env": "local",
filebeat_1_d541bdaf93da  |     "githash": "bf4c562fc2c47ecf158b7a471446200cd91140c9"
filebeat_1_d541bdaf93da  |   },
filebeat_1_d541bdaf93da  |   "host": {
filebeat_1_d541bdaf93da  |     "name": "fe62ee45f705"
filebeat_1_d541bdaf93da  |   }
filebeat_1_d541bdaf93da  | }

This breaks a number of things, for instance:

• the add_docker_metadata processor -- I worked around that by matching on container.short_id instead (setup not pictured), which my app is setting to the same value as it's trying to set host.name
• the nifty links on Kibana to switch between APM traces, metrics, and host logs; since the logs are all annotated with the beat's hostname they don't match the rest

Is this a known bug in Filebeat? If not, what can I do to work around this behavior?