Convert Multiple Fields - Ingest Node - Convert Processor

Elastic Stack Elasticsearch
1

I want to configure an ingest node to convert multiple fields coming through a pipeline but when I define more than one "convert" processor, I get a duplicate key error in the Kibana console and they overwrite each other. I've done some research but can't seem to find any evidence of this problem anywhere. I can't imagine that a given pipeline is only allowed to convert one field but the documentation only has a single field example. Can anyone help me out?

Thanks,
Brandon

Here's an example of my desired ingest pipeline:

PUT _ingest/pipeline/arduino_temp_parse
{
    "description": "parse the data coming from the arduino sensors",
    "processors": [
        {
            "grok": {
                "field": "message",
                "patterns": [
                    "%{GREEDYDATA:degrees_c}C %{GREEDYDATA:degrees_f}F %{GREEDYDATA:humidity}"
                ]
            },
            "convert": {
                "field": "degrees_c",
                "type": "integer"
            },
            "convert": {
                "field": "degrees_f",
                "type": "double"
            },
            "convert": {
                "field": "humidity",
                "type": "integer"
            }
        }
    ]
}
2

After looking at some more examples I found in blog posts, I figured it out. I needed another layer of curly brackets around each of my processors. it worked before adding the extra brackets with a single convert but the brackets were necessary to avoid the duplicate key problem. Here is my new working ingest pipeline.

{
    "arduino_temp_parse": {
        "description": "parse the data coming from the arduino sensors",
        "processors": [
            {
                "grok": {
                    "field": "message",
                    "patterns": [
                        "%{GREEDYDATA:degrees_c}C %{GREEDYDATA:degrees_f}F %{GREEDYDATA:humidity}"
                    ]
                }
            },
            {
                "convert": {
                    "field": "degrees_c",
                    "type": "integer"
                }
            },
            {
                "convert": {
                    "field": "degrees_f",
                    "type": "float"
                }
            },
            {
                "convert": {
                    "field": "humidity",
                    "type": "integer"
                }
            }
        ]
    }
}

(I also had to use float instead of double)

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.