Logstash grok pattern COMBINEDAPACHELOG field name collision with latest filebeat 7.2.0 client and logstash useragent plugin

Hi,

there seems to be issues with logstash grok pattern COMBINEDAPACHELOG with latest filebeat 7.2.0 client and logstash useragent plugin.

I am using grok pattern COMBINEDAPACHELOG which translates to %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}.

Latest filebeat client is sending agent field with following properties:
agent.ephemeral_id
agent.hostname
agent.id
agent.type
agent.version

I am using logstash-filter-useragent with following configuration:

if ([agent]) {
useragent {
source => "agent"
target => "user_agent"
remove_field => "agent"
}
}

Filebeat clients overwrites agent field from grok pattern COMBINEDAPACHELOG with filebeat agent field which results with errors in logstash:

[ERROR][logstash.filters.useragent] Uknown error while parsing user agent data {:exception=>#<TypeError: cannot convert instance of class org.jruby.RubyHash to class java.lang.String>, :field=>"agent", :event=>#LogStash::Event:0x6b358a65}

One solution of the problem would be changing COMBINEDAPACHELOG grok pattern to COMMONAPACHELOG %{QS:referrer} %{QS:someotherfield} and modify useragent configuration.

Is there any configuration option on filebeat client - change field agent to something else ?

Hello @iuuuuan,

The grok pattern that you are referring is located in Logstash? You could use the mutate the event before applying the grok?

Yes it is located in Logstash.

I have changed COMBINEDAPACHELOG to COMMONAPACHELOG %{QS:referrer} %{QS:browser} and modified useragent section to:

if ([browser]) {
useragent {
source => "browser"
target => "user_agent"
remove_field => "browser"
}
}

This configuration is working.

Base logstash grok patterns are described on https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns.

Ok I could also use mutate with rename

mutate {
rename => ["agent", "filebeat_agent" ]
}

But the basic problem still persists - filebeat agent creates field with name agent and grok pattern COMBINEDAPACHELOG has same field name as filebeat agent.

I understand the conflict, glad there is a workaround the problem. But I agree we could make a bit of work to make sure that fields extracted from grok pattern doesn't do any conflict with fields defined by filebeat.

I suggest you to create an issue on the logstash pattern repository https://github.com/logstash-plugins/logstash-patterns-core

But keep in mind I am not sure that we can solve all theses conflict.

What if I create issue for filebeat instead ? Grok pattern COMBINEDAPACHELOG is used for a long time now, filebeat agent field feature has been added recently ? Or maybe feature request for filebeat - add configuration option for filebeat field name. According to https://github.com/elastic/beats/blob/master/CHANGELOG.asciidoc agent field name has been changed from beat.name to agent.type, beat.hostname to agent.hostname and beat.version to agent.version version 7.0.0-alpha1.