Logstash grok pattern COMBINEDAPACHELOG field name collision with latest filebeat 7.2.0 client and logstash useragent plugin

Elastic Stack Beats
1

Hi,

there seems to be issues with logstash grok pattern COMBINEDAPACHELOG with latest filebeat 7.2.0 client and logstash useragent plugin.

I am using grok pattern COMBINEDAPACHELOG which translates to %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}.

Latest filebeat client is sending agent field with following properties:
agent.ephemeral_id
agent.hostname
agent.id
agent.type
agent.version

I am using logstash-filter-useragent with following configuration:

if ([agent]) {
useragent {
source => "agent"
target => "user_agent"
remove_field => "agent"
}
}

Filebeat clients overwrites agent field from grok pattern COMBINEDAPACHELOG with filebeat agent field which results with errors in logstash:

[ERROR][logstash.filters.useragent] Uknown error while parsing user agent data {:exception=>#<TypeError: cannot convert instance of class org.jruby.RubyHash to class java.lang.String>, :field=>"agent", :event=>#LogStash::Event:0x6b358a65}

One solution of the problem would be changing COMBINEDAPACHELOG grok pattern to COMMONAPACHELOG %{QS:referrer} %{QS:someotherfield} and modify useragent configuration.

Is there any configuration option on filebeat client - change field agent to something else ?

2

Hello @iuuuuan,

The grok pattern that you are referring is located in Logstash? You could use the mutate the event before applying the grok?

3

Yes it is located in Logstash.

I have changed COMBINEDAPACHELOG to COMMONAPACHELOG %{QS:referrer} %{QS:browser} and modified useragent section to:

if ([browser]) {
useragent {
source => "browser"
target => "user_agent"
remove_field => "browser"
}
}

This configuration is working.

Base logstash grok patterns are described on https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns.

Ok I could also use mutate with rename

mutate {
rename => ["agent", "filebeat_agent" ]
}

But the basic problem still persists - filebeat agent creates field with name agent and grok pattern COMBINEDAPACHELOG has same field name as filebeat agent.

4

I understand the conflict, glad there is a workaround the problem. But I agree we could make a bit of work to make sure that fields extracted from grok pattern doesn't do any conflict with fields defined by filebeat.

I suggest you to create an issue on the logstash pattern repository https://github.com/logstash-plugins/logstash-patterns-core

But keep in mind I am not sure that we can solve all theses conflict.

5

What if I create issue for filebeat instead ? Grok pattern COMBINEDAPACHELOG is used for a long time now, filebeat agent field feature has been added recently ? Or maybe feature request for filebeat - add configuration option for filebeat field name. According to https://github.com/elastic/beats/blob/master/CHANGELOG.asciidoc agent field name has been changed from beat.name to agent.type, beat.hostname to agent.hostname and beat.version to agent.version version 7.0.0-alpha1.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.