Hey @Mario_Castro
Thanks for your response. It turns out, add_host_metadata wasn't the culprit after all. As you described, it's only enabled by default via the reference yml file, and since I'm using a config that doesn't include that processor, it's not what's actually overwriting the host field in my case, it must be something else.
He also mentioned that the host.name will still be written even if you disable add_host_metadata
That's exactly what I'm seeing ("host": {"name":"dc79b87d3f81"})
and that it's not recommended to use a processor to remove it.
Well, it doesn't seem that I have a lot of options here. I absolutely, positively need the payload data from my event's host field (contains the remove client IP in structured (JSON) web server access logs).
And it doesn't seem that there's a way to rename my own host field to something else before filebeat overwrites it (in order to preserve its contents). I would be totally fine with renaming my field to something like client_ip, but I can't change the structure of the incoming JSON files that filebeat reads - and in those, the field is called host unfortunately.
I've did read the breaking changes section you linked (should've mentioned that in my original post, sorry), but unfortunately it doesn't cover my case at all - how to preserve data from an actual payload field called host, when it conflicts with the host fields from ECS / filebeat.
When your workmate says it's not recommended to use a processor to remove it, could you please ask him to elaborate on that a bit, if you get a chance? What kinds of issues could I be facing if I do that?
Because I managed to find a workaround that kinda does the trick for me.
The problem seems to be that json.overwrite_keys: true doesn't work to override the host field, but using the decode_json_fields processor (instead of the json reader in the prospector/input) and enabling overwrite_keys for it actually does.
Can you link me to the Docker image you are using?
I managed to reproduce the behavior without the Docker image (see below). But I was using the filebeat 6.5.4 image ("gave up" on 6.3 and tried to investigate & reproduce the issue directly with 6.5). But in the end I didn't really see a difference in behavior between docker vs. running FB on a real Centos machine.
So, to summarize with a minimal testcase (on a Centos 7.4 machine, taking docker and logstash out of the mix):
The logfile contains events (one JSON object per line) with the following shape:
{"host": "192.168.10.10", "user": "john.doe"}
This config does not work:
filebeat.inputs:
- type: log
paths:
- /var/log/sample-*.log
json.keys_under_root: true
json.add_error_key: true
json.overwrite_keys: true
output.file:
path: "/var/log"
filename: filebeat.out.log
codec.json:
pretty: true
It results in the following output (fb.local is the machine's hostname, "192.168.10.10" is the value that should have been preserved for the host field according to json.overwrite_keys):
{
"@timestamp": "2018-12-28T10:11:34.719Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.5.4"
},
"source": "/var/log/sample-2018-12-28 11:11:32+01:00.log",
"offset": 0,
"json": {
"host": "192.168.10.10",
"user": "john.doe"
},
"prospector": {
"type": "log"
},
"input": {
"type": "log"
},
"host": {
"name": "fb.local"
},
"beat": {
"name": "fb.local",
"hostname": "fb.local",
"version": "6.5.4"
}
}
The following config works however:
filebeat.inputs:
- type: log
paths:
- /var/log/sample-*.log
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 2
target: ""
overwrite_keys: true
- drop_fields:
fields: ["message"]
output.file:
path: "/var/log"
filename: filebeat.out.log
codec.json:
pretty: true
Resulting in this output, where the host field's content is preserved:
{
"@timestamp": "2018-12-28T10:07:39.306Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.5.4"
},
"prospector": {
"type": "log"
},
"input": {
"type": "log"
},
"host": "192.168.10.10",
"beat": {
"name": "fb.local",
"hostname": "fb.local",
"version": "6.5.4"
},
"user": "john.doe",
"source": "/var/log/sample-2018-12-28 11:07:32+01:00.log",
"offset": 0
}