Convert type of custom fields added via metricbeat processors

Elastic Stack Beats
1

I have added the following processor in system.yml module of metricbeat.
These custom fields flow to elastisearch as objects, but I am looking to pass them as strings.
I tried using the "convert" processor without any luck. Is there a way to achieve this?

  processors:
  - add_fields:
      target: ''
      fields:
        module: Infra
        sub_module: System
2

Hi!

Could you share the resulted events? According to the docs
to store the fields as top-level fields, set target: '', so what do you want to see as string and it isn't?

C.

3

Hi @ChrsMark,
Let me set the context properly. As in the initial question, I am looking to add the fields "module" and "sub_module" as custom text fields via Metricbeat. I have an existing index which is created by Logstash that has these two fields as text. I have made appropriate changes to Metricbeat to align with the index name created by Logstash so that both indices come under a single index pattern, which would eventually help me in building visualizations as per my needs.
I have attached the screenshot from Kibana about this conflict for more clarity. In the attachment, indices of the format "or_field_reserve-XXXXXX" are created by Logstash and "or_field_reserve_metric-XXXX.XX.XX-XXXXXX are created by Metricbeat. Below is the mapping of the field "module" from Logstash and Metricbeat.

Logstash:

"module" : {
 "type" : "text",
 "norms" : false,
 "fields" : {
   "keyword" : {
      "type" : "keyword",
      "ignore_above" : 256
   }
   }
}

Metricbeat:

"module": {
"type": "keyword",
"ignore_above": 1024
}

Metricbeat_add_fields_error
Metricbeat_add_fields_error1048×569 59.6 KB

4

Hi!

I tried with your example:

processors:
  - add_fields:
      target: ''
      fields:
        module: Infra
        sub_module: System

and here is what I see:

Screenshot 2020-02-03 at 11.10.48
Screenshot 2020-02-03 at 11.10.481082×245 14 KB

Consequently the processor itself works as expected, since it stores the fields as strings. So I guess that in your case, the thing that you have Logstash indexing the same field causes this confusion in ES.

Since it tends to be an Elasticsearch specific question I would suggest that you could point it to the respective Elasticsearch forum? wdyt?

5

Hi @ChrsMark,
I have applied a workaround for now by updating the mapping in the index template created by metricbeat. By adding the below in the metricbeat mapping, I was able to align the fields flowing from both sources. Could you please share the mapping for the fields created by your replication? You can find it under "Management-> Index Management-->Index templates--> --> Mappings or from the dev tools using query in Kibana.

"module" : {
  "type" : "text",
  "norms" : false,
  "fields" : {
	"keyword" : {
	  "type" : "keyword",
	  "ignore_above" : 256
	}
  }
},
"sub_module" : {
  "type" : "text",
  "norms" : false,
  "fields" : {
	"keyword" : {
	  "type" : "keyword",
	  "ignore_above" : 256
	}
  }
6

Hey!

Here it is:

    "module" : {
        "type" : "keyword",
        "ignore_above" : 1024
    },
    "sub_module" : {
        "type" : "keyword",
        "ignore_above" : 1024
    },
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.