# Filebeat Hints-based Autodiscovery: how to use copy\_fields or rename processor in Kubernetes annotation? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 20, 2020, 7:22am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476 "2020-05-20T07:22:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Jan\_Malcomess](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_malcomess/32/45243_2.png) [@Jan\_Malcomess](https://discuss.elastic.co/u/Jan_Malcomess)\ **Post date:** [May 20, 2020, 7:22am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/1 "2020-05-20T07:22:39Z")
Hi there, im trying to use hints-based autodiscovery in our Openshift/Kubernetes environment to dissect the logs of our Springboot-based microservices (Filbeat 7.7.0). So far, dissecting the message and parsing the timestamp are working (NO thanks to the abysmal documenation of the Filebeat dissect processor, I might add). What I now want to do is to replace the 'message' field with the extracted message I got using the dissect processor. In order to do so, I need to use the copy\_fields or rename\_fields processor, e.g., like this: ```auto co.elastic.logs/processors.1.add_fields.target: 'service' co.elastic.logs/processors.1.add_fields.fields.name: '${APP_SERVICE_NAME}' co.elastic.logs/multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' co.elastic.logs/multiline.negate: 'true' co.elastic.logs/multiline.match: 'after' co.elastic.logs/processors.2.rename.fields.from: message co.elastic.logs/processors.2.rename.fields.to: undissected_message co.elastic.logs/processors.3.dissect.tokenizer: >- %{+datetime} %{+datetime->} %{log.level} %{?pid} --- [%{process.thread.name}] %{log.logger->} : %{message} co.elastic.logs/processors.3.dissect.field: 'undissected_message' co.elastic.logs/processors.3.dissect.target_prefix: '' co.elastic.logs/processors.4.timestamp.field: 'datetime' co.elastic.logs/processors.4.timestamp.layouts: '2006-01-02 15:04:05.000' co.elastic.logs/processors.5.drop_fields.fields: 'datetime, undissected_message' ``` However, that fails because the 'rename' processor expects to be passed an array of objects. So, I tried this: ```auto co.elastic.logs/processors.2.rename.fields: - from: message to: undissected_message ``` That fails, because K8s only allows Strings in annotations, not objects. I then tried this: ```auto co.elastic.logs/processors.2.rename.fields: '{[{from:message, to:undissected_message}]}' ``` Which failed because Filebeat complained that it expected an object and not a string. How am I supposed to configure the copy\_fields- and rename-processors using annotations? If it's not possible, is this a bug that can be fixed? If it's not, shouldn't the documentation clearly warn people to not trying to use those in hints-based autodiscovery so they are spared of wasting hours on trying to get this to work? Maybe someone has had more luck than me using this... Big thanks to whoever provides a useful answer 🙂 ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [May 20, 2020, 10:30am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/2 "2020-05-20T10:30:44Z")
Did you try this format: `co.elastic.logs/processors.2.rename.fields.1.from: message` ? ---
**Author:** ![Jan\_Malcomess](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_malcomess/32/45243_2.png) [@Jan\_Malcomess](https://discuss.elastic.co/u/Jan_Malcomess)\ **Post date:** [May 20, 2020, 11:07am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/3 "2020-05-20T11:07:37Z")
Yes, with the same result: Filebeat complains that it wants an object, but got a string ☹ ---
**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\ **Post date:** [May 20, 2020, 1:36pm UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/4 "2020-05-20T13:36:40Z")
You might have actually encountered a real bug. Could you please paste here the output you received? EDIT: You can try to use also: `co.elastic.logs/processors.2.rename.fields: '[{from:message, to:undissected_message}]'` (it's an array). ---
**Author:** ![Jan\_Malcomess](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_malcomess/32/45243_2.png) [@Jan\_Malcomess](https://discuss.elastic.co/u/Jan_Malcomess)\ **Post date:** [May 22, 2020, 3:43am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/5 "2020-05-22T03:43:45Z")
Ok, I have to retract my initial response: the error to the attempt of `co.elastic.logs/processors.2.rename.fields.1.from: message` is NOT Filebeat complaing that it expected an error but got a string. Instead, Filebeat seems to process the configuration, but then logs an error in the field 'error.message': `Failed to rename fields in processor: could not fetch value for key: , Error: key not found` Seems as though the configured name of the key (here: 'message') is ignored (I tried it also with some other fields to the same result). ---
**Author:** ![Jan\_Malcomess](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_malcomess/32/45243_2.png) [@Jan\_Malcomess](https://discuss.elastic.co/u/Jan_Malcomess)\ **Post date:** [May 22, 2020, 3:49am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/6 "2020-05-22T03:49:33Z")
As for the array attempt, I tried various permutations, all with the same result: Filebeat throws an error and no log message gets pushed to Elasticsearch. I tried: ```auto co.elastic.logs/processors.2.rename.fields: '[{from:message, to:undissected_message}]' ``` ```auto co.elastic.logs/processors.2.rename.fields: '[{from:"message", to:"undissected_message"}]' ``` and ```auto co.elastic.logs/processors.2.rename.fields: '[{"from":"message", "to":"undissected_message"}]' ``` and the error logged by Filebeat is: ```auto 2020-05-22T03:45:57.813Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: failed to unpack the rename configuration: required 'object', but found 'string' in field 'processors.1.rename.fields' ``` ---
**Author:** ![Jan\_Malcomess](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_malcomess/32/45243_2.png) [@Jan\_Malcomess](https://discuss.elastic.co/u/Jan_Malcomess)\ **Post date:** [May 22, 2020, 3:53am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/7 "2020-05-22T03:53:47Z")
Something I only just saw: the error message logged by Filebeat states 'processors. **1**.rename.fields' even though I have defined that processor 'co.elastic.logs/processors. **2**.rename.fields' ---
**Author:** ![Felix\_Roessel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felix_roessel/32/41623_2.png) [@Felix\_Roessel](https://discuss.elastic.co/u/Felix_Roessel)\ **Post date:** [May 25, 2020, 6:48am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/8 "2020-05-25T06:48:58Z")
Is there a specific reason not using the ingest processors of ES for that? [https://www.elastic.co/guide/en/elasticsearch/reference/master/dissect-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/dissect-processor.html) Would be easier to solve the problem I think. And you can change it on the fly via Kibana whenever you need. ---
**Author:** ![Jan\_Malcomess](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan_malcomess/32/45243_2.png) [@Jan\_Malcomess](https://discuss.elastic.co/u/Jan_Malcomess)\ **Post date:** [May 25, 2020, 7:45am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/9 "2020-05-25T07:45:23Z")
We certainly could do that, but: We have a very heterogenous landscape with different technologies and log layouts. Using the hints-based autodiscover, each team can customize the logscraping to their needs relatively easily without us having to manage various ingest pipelines. The ability to use the rename- or copy\_fields-processor in the hints-based autodiscover configuration on Kubernetes/Openshift is a nice-to-have to us. If these cannot be used, then this is not a problem to us. However, the documentation of hints-based autodiscover should be adapted to state that those cannot be used. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2020, 7:45am UTC](https://discuss.elastic.co/t/filebeat-hints-based-autodiscovery-how-to-use-copy-fields-or-rename-processor-in-kubernetes-annotation/233476/10 "2020-06-22T07:45:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.