Filebeat tags not applied, if JSON event already has a tags field?

[uschtwill (Wilhelm Uschtrin)]

I have spent quite some time trying to add some tags to Kibana logs yesterday - futilely.

The Kibana logs are in JSON and already contain a "tags" field. The way it looks, Filebeat (maybe beats in general?) chooses not to add tags to an event, if the event has a pre-existing "tags" field - regardless of wether the tags are added via the prospector or "generally".

Is this possible?

[steffens (Steffen Siering)]

Can you share your config? Beats expects tags to be an array of strings.

[uschtwill (Wilhelm Uschtrin)]

Sure:

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    [{"paths": ["/var/log/*.log", "/var/log/redacted/kibana.json"], "tags": ["filebeat", "filebeat-dev"], "ignore_older": "24h", "encoding": "utf-8", "field": {"type": "filebeatlog"}, "registry_file": "/var/lib/filebeat/registry", "input_type": "log"}]

[steffens (Steffen Siering)]

You also have a sample log event? The tags field must be an array of strings. Checking the source code there might indeed be a problem with the JSON parser not representing the array with the assumed types.

[uschtwill (Wilhelm Uschtrin)]

They're really just normal Kibana JSON logs, and tags also contains proper strings:

{"type":"log","@timestamp":"2017-08-22T09:11:11Z","tags":["status","plugin:kibana@5.4.0","info"],"pid":17341,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-08-22T09:11:11Z","tags":["status","plugin:elasticsearch@5.4.0","info"],"pid":17341,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}

[steffens (Steffen Siering)]

Thanks. I'd say this is a bug in the parsing/normalization code in filebeat. The post-processing should try to convert the tags value to it's proper type.

I filed this bug report. Feel free to add any context/config/logs/events that might help. Btw. it's tagged for Hacktoberfest if you want to give it a shot

[uschtwill (Wilhelm Uschtrin)]

Thanks, for following up on this, you're welcome.