How to parse field named "host" (or any other field which is reserved by filebeat) in JSON structured log?

[elasticheart]

Hi,

I am trying to publish messages into Kafka using filebeat 6.3.0. My log files has single line, JSON structured logs, like;

{"f1":"data1","host":"server1","timestamp":"2018-09-10T12:33:15.878+0000","f2":"data2","f3":1234,"f4":"data4","server":"192.168.0.1","f5":"data5","f6":"1","f7":"data7","f8":"data8","f9":"data9","f10":false}

I am using filebeat JSON options like;

json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: false

As you can see, I have a host field in my logs. But, when it comes to kafka / output, it comes like;

"host": {
	"name": "linuxbox"
},

which is the name of my machine. But I want it to be as shown in the log file. How can I fix this?

Thanks.

[jsoriano (Jaime Soriano)]

Hi @elasticheart,

Indeed host is a field managed by each Beat, so it is not recommended to use it in your custom events. Would json.keys_under_root: false be an option for you?

If you need to keep the fields in the top level in any case there is a way to circumvent the override of reserved fields taking advantage of current implementation.

In the output pipeline, input-specific processors are executed first, then "builtin" fields like host.name are added, and finally global processors are executed. So you can rename fields so they are not overwritten by the builtin fields, and then if you want you can overwrite the builtin field.

For that you need to add a processor to prevent your value to be overwritten, e.g:

- type: log
  ...
  processors:
  - rename:
      fields:
      - from: "host"
        to: "host.srcname"

With this your field will be kept on host.srcname. If you want to override the field added by filebeat, you can add this top-level configuration:

processors:
- drop_fields:
    when:
      has_fields: ['host.srcname']
    fields: ['host.name']
- rename:
    when:
      has_fields: ['host.srcname']
    fields:
    - from: 'host.srcname'
      to: 'host.name'

But remember that this option depends on current implementation, and the way these builtin fields are managed can change in the future.