Question about logstash output plugin 7.9 and ecs

Elastic Stack Logstash
1

I am not understanding the ecs changes in the logstash elasticsearch output plugin version 7.9 ecs compatibility.
I have an issue when I try to index an event that has an object mapping for host.
object mapping for [host] tried to parse field [host] as object
I know this is a common error in logstash
My question is does logstash output plugin version 7.10 fix this problem or is this something I need to do in my logstash pipelines?

From 7.10 documentation
This plugin will persist events to Elasticsearch in the shape produced by your pipeline, and cannot be used to re-shape the event structure into a shape that complies with ECS. To produce events that fully comply with ECS, you will need to populate ECS-defined fields throughout your pipeline definition.

Can anyone interpret this language to a more layman's terms?
does this mean that if filebeat sets host to object any event with host as non-object will still be rejected?

2

@gerbdla / David

I don't think this is so much an output plugin issue - (judging from behavior, not actually looking at the code) the issue is that logstash will in certain circumstances automatically include a [host] field (e.g. the sending system for syslog).

Basically if the index template for the index you are writing to is based on ECS it will include host as an object (host.name, host.ip, etc.) which precludes using host as a flat field.

I typically do the following:
mutate { rename => { "host" => "[host][ip]" }}
Or for observers (e.g. firewall logs) I'll rename it to [observer][ip]

I believe there are some PRs in the logstash github repo regarding this, but I haven't really dug into it.

In case you aren't familiar with the differences (I had no idea until someone pointed it out) Logstash and Elasticsearch represent objects differently - e.g. [host][ip] in Logstash is the same as host.ip in Elasticsearch.

The docs are basically saying:
the Elasticsearch output simply sends what you give it, to be ECS compliant its up to you to properly name fields in your logstash configs.

Hope this helps!
/d

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.