Copy a value from one field to other if a field exists by using ingestnode pipeline

Elastic Stack Elasticsearch
1

Hi Team,
I want to create a new field called kubernetes.pod.name if fields called prometheus.labels.pod exists in the logs. I found out that from the set processor I could copy the value which is present in prometheus.labels.pod to a new field kubernetes.pod.name but I need to do this conditionally as the pod name keeps on changing.

How do i set a condition such that if field prometheus.labels.pod exists then only I need to add a new field called kubernetes.pod.name (both has the same value)

ctx.prometheus?.labels?.namespace== "name_of_namespace"

could be do similarly can we do

ctx.prometheus?.labels?.pod== "*"

to check if this field exists or not ?

3

See if this works for you.

POST /_ingest/pipeline/_simulate?verbose=true
{
  "pipeline" :
  {
    "processors": [
      {
        "set" : {
          "field" : "kubernetes.pod.name",
          "value" : "{{prometheus.labels.pod}}",
          "if" : "ctx.containsKey('prometheus.labels.pod')"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "prometheus.labels.pod": "test"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}
4

Hi @aaron-nimocks ,

Nop not working. What's your view on using this in set processor ?

ctx?.prometheus?.labels?.pod!=null

5

What exactly does your data look like? The above example works if it's formatted like this.

      "_source": {
        "prometheus.labels.pod": "test"
      }

I don't think ctx?.prometheus?.labels?.pod!=null will work. The ctx.containsKey('prometheus.labels.pod') checks to see if the field exists and I think that's what you are looking for.

6

When i use containsKey it's skipping that set condition. Data looks like this :

"prometheus.labels.pod": [
"test"
]

7

I don't know the answer yet, might require another processor. But the issue is you have an array based on your last message. I thought it was a string before.

So if you run the below in dev tools you will see one of the 2 messages will be true on the condition and it will create a new field but I am not sure you can target the first array element. Is there always only one value in the array? If there is 2 values then which do you use?

"kubernetes" : {
 "pod" : {
  "name" : "{0=test}"
 }
}

POST /_ingest/pipeline/_simulate?verbose=true
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "kubernetes.pod.name",
          "value": "{{prometheus.labels.pod}}",
          "if": "ctx.containsKey('prometheus.labels.pod')"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "prometheus.labels.pod": [
          "test"
        ]
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}
8

There will be always 1 value and if that field exists I have to create another field kubernetes.pod.name. It won't have more than 1 value.

Any idea why the below would be incorrect ?

ctx?.prometheus?.labels?.pod!=null

9

Because it's an array. There is a big difference between.

"_source": {
 "prometheus.labels.pod": [
  "test"
 ]
}

and

"_source": {
 "prometheus.labels.pod": "test"
}

Also if you were just checking to see if the field was null or empty then records without that field would return true and it would create a new field with no value.

10

I am so sorry it's a string i just check again kibana says this field is string and I checked my data in json format. then could see

"_source": {
"prometheus": {
  "query": {
    "count": 142
  },
  "labels": {
    "pod": "test"
   
   }
 }
}
11

On that note what you said should work.

POST /_ingest/pipeline/_simulate?verbose=true
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "kubernetes.pod.name",
          "value": "{{prometheus.labels.pod}}",
          "if": "ctx?.prometheus?.labels?.pod != null"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "prometheus": {
          "query": {
            "count": 142
          },
          "labels": {
            "pod": "test"
          }
        }
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    }
  ]
}
12

Yes, I tested for different cases and works as expected. Thanks a lot for your time and analysis. Much appreciated!