Grok ecs_compatibility option

Elastic Stack Logstash
1

Hey all,

I noticed that my grok filter that uses the COMBINEDAPACHELOG pattern (logstash-patterns-core/patterns/ecs-v1/httpd at main · logstash-plugins/logstash-patterns-core · GitHub) doesn't actually return the [user][name] and [apache][access][user][identity] fields even though they're in the pattern.

Under the ecs_compatibility option on the grok documentation page there's a statement that I don't understand that maybe explains this?

Controls this plugin’s compatibility with the Elastic Common Schema (ECS). The value of this setting affects extracted event field names when a composite pattern (such as HTTPD_COMMONLOG) is matched.

Can somebody clarify what "The value of this setting affects extracted event field names when a composite pattern is matched" means? Does this explain why the [user][name] and [apache][access][user][identity] fields are missing?

Thanks,
Frans

2

The ecs_compatibility option determines whether it uses this set of patterns, or this one.

The part about composite patterns is, strictly speaking, true, but it doesn't really add any value to the sentence. The settings affects the field names.

1 Like
3

Thanks @Badger !

I now see the patterns for each part of the log line are (?:-|%{PATTERN:variable}) which I guess means that if the value in the log line is a literal - then the second part of that expression isn't applied and hence variable isn't set. I guess that explains why I'm not getting the variable in the output. Then I just need to figure out why nginx (which is creating the logs in my case) is printing - instead of the actual username.