Filebeat Hints-based Autodiscovery: how to use copy_fields or rename processor in Kubernetes annotation?

[Jan_Malcomess]

Hi there,

im trying to use hints-based autodiscovery in our Openshift/Kubernetes environment to dissect the logs of our Springboot-based microservices (Filbeat 7.7.0). So far, dissecting the message and parsing the timestamp are working (NO thanks to the abysmal documenation of the Filebeat dissect processor, I might add).
What I now want to do is to replace the 'message' field with the extracted message I got using the dissect processor. In order to do so, I need to use the copy_fields or rename_fields processor, e.g., like this:

    co.elastic.logs/processors.1.add_fields.target: 'service'
    co.elastic.logs/processors.1.add_fields.fields.name: '${APP_SERVICE_NAME}' 
    co.elastic.logs/multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' 
    co.elastic.logs/multiline.negate: 'true'
    co.elastic.logs/multiline.match: 'after'
    co.elastic.logs/processors.2.rename.fields.from: message
    co.elastic.logs/processors.2.rename.fields.to: undissected_message 
    co.elastic.logs/processors.3.dissect.tokenizer: >- 
      %{+datetime} %{+datetime->} %{log.level} %{?pid} --- [%{process.thread.name}] %{log.logger->} : %{message}
    co.elastic.logs/processors.3.dissect.field: 'undissected_message'
    co.elastic.logs/processors.3.dissect.target_prefix: ''
    co.elastic.logs/processors.4.timestamp.field: 'datetime'
    co.elastic.logs/processors.4.timestamp.layouts: '2006-01-02 15:04:05.000'
    co.elastic.logs/processors.5.drop_fields.fields: 'datetime, undissected_message'

However, that fails because the 'rename' processor expects to be passed an array of objects. So, I tried this:

    co.elastic.logs/processors.2.rename.fields:
      - from: message
        to: undissected_message

That fails, because K8s only allows Strings in annotations, not objects.
I then tried this:

co.elastic.logs/processors.2.rename.fields: '{[{from:message, to:undissected_message}]}'

Which failed because Filebeat complained that it expected an object and not a string.

How am I supposed to configure the copy_fields- and rename-processors using annotations?
If it's not possible, is this a bug that can be fixed?
If it's not, shouldn't the documentation clearly warn people to not trying to use those in hints-based autodiscovery so they are spared of wasting hours on trying to get this to work?

Maybe someone has had more luck than me using this...

Big thanks to whoever provides a useful answer

[mtojek (Marcin Tojek)]

Did you try this format:

co.elastic.logs/processors.2.rename.fields.1.from: message ?

[Jan_Malcomess]

Yes, with the same result: Filebeat complains that it wants an object, but got a string

[mtojek (Marcin Tojek)]

You might have actually encountered a real bug. Could you please paste here the output you received?

EDIT:

You can try to use also:

co.elastic.logs/processors.2.rename.fields: '[{from:message, to:undissected_message}]' (it's an array).

[Jan_Malcomess]

Ok, I have to retract my initial response: the error to the attempt of

co.elastic.logs/processors.2.rename.fields.1.from: message

is NOT Filebeat complaing that it expected an error but got a string. Instead, Filebeat seems to process the configuration, but then logs an error in the field 'error.message':

Failed to rename fields in processor: could not fetch value for key: , Error: key not found

Seems as though the configured name of the key (here: 'message') is ignored (I tried it also with some other fields to the same result).

[Jan_Malcomess]

As for the array attempt, I tried various permutations, all with the same result: Filebeat throws an error and no log message gets pushed to Elasticsearch. I tried:

co.elastic.logs/processors.2.rename.fields: '[{from:message, to:undissected_message}]' 

co.elastic.logs/processors.2.rename.fields: '[{from:"message", to:"undissected_message"}]' 

and

co.elastic.logs/processors.2.rename.fields: '[{"from":"message", "to":"undissected_message"}]' 

and the error logged by Filebeat is:

2020-05-22T03:45:57.813Z	ERROR	[autodiscover]	cfgfile/list.go:96	Error creating runner from config: failed to unpack the rename configuration: required 'object', but found 'string' in field 'processors.1.rename.fields'

[Jan_Malcomess]

Something I only just saw: the error message logged by Filebeat states 'processors.1.rename.fields' even though I have defined that processor 'co.elastic.logs/processors.2.rename.fields'

[Felix_Roessel]

Is there a specific reason not using the ingest processors of ES for that?
https://www.elastic.co/guide/en/elasticsearch/reference/master/dissect-processor.html

Would be easier to solve the problem I think. And you can change it on the fly via Kibana whenever you need.

[Jan_Malcomess]

We certainly could do that, but:
We have a very heterogenous landscape with different technologies and log layouts. Using the hints-based autodiscover, each team can customize the logscraping to their needs relatively easily without us having to manage various ingest pipelines.
The ability to use the rename- or copy_fields-processor in the hints-based autodiscover configuration on Kubernetes/Openshift is a nice-to-have to us. If these cannot be used, then this is not a problem to us.
However, the documentation of hints-based autodiscover should be adapted to state that those cannot be used.