# Add new field to index based on maths calculation from other fields in the same index

**URL:** <https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571>\
**Category:** Elasticsearch\
**Created:** [July 4, 2023, 2:04pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571 "2023-07-04T14:04:47Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 4, 2023, 2:04pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/1 "2023-07-04T14:04:47Z")

</div>

Hi,

I use elastic-agent on EKS with kubernetes integration.

One of the field such as kubernetes.volume.fs.used.pct in the index provides incorrect values

I was able to get the correct value using the following formula:

> subtract(100, multiply(average(kubernetes.volume.fs.available.bytes) / max(kubernetes.volume.fs.capacity.bytes), 100))

Based on that new metric, I would like to create alert when the used pod volume is above 80%.  
I have added the following processor configuration to ingest pipeline

`metrics-kubernetes.volume@custom`

which should add new filed to the index:

> ctx['kubernetes.volume.fs.used.custom.pct'] = 100 - (ctx['kubernetes.volume.fs.available.bytes'] / ctx['kubernetes.volume.fs.capacity.bytes']) \* 100

It doesn't work as the new filed is empty and upon checking now - it doesn't exist at all.

The question is - how to add new field to index based on maths calculation from other fields in the same index as required for alerting?

I assume that would have to be done later than ingest pipeline.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 4, 2023, 2:20pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/2 "2023-07-04T14:20:04Z")

</div>

Hi @patcan Welcome to the community!

You need to share your actual ingest pipeline you can get it from Kibana -\> Dev Tools with

`GET _ingest/pipeline/metrics-kubernetes.volume@custom`

> [@patcan](#):
>
> ctx['kubernetes.volume.fs.used.custom.pct'] = 100 - (ctx['kubernetes.volume.fs.available.bytes'] / ctx['kubernetes.volume.fs.capacity.bytes']) \* 100

That is the wrong syntax for accessing fields in an ingest pipeline ...  
Please look / read the docs [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#access-source-fields) there are different syntax for conditions vs accessing the field data (which is what you are doing)

> Several processor parameters support [Mustache](https://mustache.github.io/) template snippets. To access field values in a template snippet, enclose the field name in triple curly brackets:`{{{field-name}}}` . You can use template snippets to dynamically set field names.

It should look more like

```auto
"set": {
        "description": "my custom metric",
        "field": "kubernetes.volume.fs.used.custom.pct",
        "value": "100 - {{{kubernetes.volume.fs.available.bytes}}} / {{{kubernetes.volume.fs.capacity.bytes}}}) * 100"
      }

```

And really percent is a decimal so it really should be like

```auto
value": "1.0 - {{{kubernetes.volume.fs.available.bytes}}} / {{{kubernetes.volume.fs.capacity.bytes}}}) "

```

---

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 6, 2023, 6:24pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/3 "2023-07-06T18:24:41Z")

</div>

Hi @stephenb

> [@stephenb](#):
>
> GET \_ingest/pipeline/metrics-kubernetes.volume@custom

```auto
{
  "metrics-kubernetes.volume@custom": {
    "processors": [
      {
        "script": {
          "ignore_failure": true,
          "source": "ctx['kubernetes.volume.fs.used.custom.pct'] = 100 - (ctx['kubernetes.volume.fs.available.bytes'] / ctx['kubernetes.volume.fs.capacity.bytes']) * 100"
        }
      }
    ]
  }
}

```

I will try your solution. Thanks

---

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 6, 2023, 6:46pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/4 "2023-07-06T18:46:02Z")

</div>

Nope. Now I have suggested Set processor and result:

```auto
{
  "metrics-kubernetes.volume@custom": {
    "processors": [
      {
        "set": {
          "field": "kubernetes.volume.fs.used.custom.pct",
          "value": "(1.0 - {{{kubernetes.volume.fs.available.bytes}}} / {{{kubernetes.volume.fs.capacity.bytes}}})",
          "ignore_failure": true
        }
      }
    ]
  }
}

```

Getting under the kubernetes.volume.fs.used.custom.pct field value:  
`(1.0 - 9786859520 / 21462233088)`

It looks like it reads the values of the fields but doesn't perform the maths

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2023, 6:49pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/5 "2023-07-06T18:49:21Z")

</div>

~~Try taking out the double quotes~~ Wrong ! 🙂

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 6, 2023, 8:05pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/6 "2023-07-06T20:05:13Z")

</div>

@patcan

Apologies Sorry Led you astray need to use a `script` processor ... which makes sense after I thought about it ... so you were close to start...

Also _ **do not put just raw numbers into the equation** _ use `params` it is much more efficient.. in short it only needs to compile the script once

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
          "params": {
            "to_percent": 100.0,
            "one": 1.0
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "kubernetes.volume.fs.available.bytes": 9786859520,
        "kubernetes.volume.fs.capacity.bytes": 21462233088
      }
    }
  ]
}

# Results
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "kubernetes.volume.fs.capacity.bytes": 21462233088,
          "kubernetes.volume.fs.used.custom.pct": 54.39962151248816,
          "kubernetes.volume.fs.available.bytes": 9786859520
        },
        "_ingest": {
          "timestamp": "2023-07-06T20:03:34.48287543Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 7, 2023, 12:12pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/7 "2023-07-07T12:12:07Z")

</div>

Strange thing I face over here.

Coping and pasting the above POST works fine but adding the processor to ingest pipeline doesn't.  
Getting Error:

```auto
{
  "docs": [
    {
      "processor_results": [
        {
          "processor_type": "script",
          "status": "error_ignored",
          "ignored_error": {
            "error": {
              "root_cause": [
                {
                  "type": "script_exception",
                  "reason": "runtime error",
                  "script_stack": [
                    "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
                    " ^---- HERE"
                  ],
                  "script": "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
                  "lang": "painless",
                  "position": {
                    "offset": 72,
                    "start": 0,
                    "end": 189
                  }
                }
              ],
              "type": "script_exception",
              "reason": "runtime error",
              "script_stack": [
                "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
                " ^---- HERE"
              ],
              "script": "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
              "lang": "painless",
              "position": {
                "offset": 72,
                "start": 0,
                "end": 189
              },
              "caused_by": {
                "type": "null_pointer_exception",
                "reason": "Cannot invoke \"Object.getClass()\" because \"value\" is null"
              }
            }
          },
          "doc": {
            "_index": ".test",
            "_version": "-3",
            "_id": "test",
            "_source": {
              "kubernetes": {
                "volume": {
                  "name": "test",
                  "fs": {
                    "available": {
                      "bytes": 262131712
                    },
                    "used": {
                      "pct": 0.000046875,
                      "bytes": 12288
                    },
                    "inodes": {
                      "pct": 0.0000044726719742374095,
                      "count": 2012220,
                      "used": 9,
                      "free": 2012211
                    },
                    "capacity": {
                      "bytes": 262144000
                    }
                  }
                }
              }
            },
            "_ingest": {
              "pipeline": "_simulate_pipeline",
              "timestamp": "2023-07-07T12:06:45.685737624Z"
            }
          }
        }
      ]
    }
  ]
}

```

Even adding processor to ingest pipeline as JSON import doesn't help here:

```auto
  {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
          "params": {
            "to_percent": 100.0,
            "one": 1.0
          }
        }
      }
    ]
  }

```

Used document to test pipeline:

```auto
[
  {
    "_index": ".test",
    "_id": "test",
    "_source": {
      "kubernetes": {
        "volume": {
          "name": "test",
          "fs": {
            "inodes": {
              "pct": 0.0000044726719742374095,
              "count": 2012220,
              "used": 9,
              "free": 2012211
            },
            "available": {
              "bytes": 262131712
            },
            "used": {
              "pct": 0.000046875,
              "bytes": 12288
            },
            "capacity": {
              "bytes": 262144000
            }
          }
        }
      }
    }
  }  
]

```

---

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 7, 2023, 12:32pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/8 "2023-07-07T12:32:49Z")

</div>

BTW if I check the ingest pipeline - all looks good IMO

```auto
GET _ingest/pipeline/metrics-kubernetes.volume@custom

```

result:

```auto
{
  "metrics-kubernetes.volume@custom": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": "ctx['kubernetes.volume.fs.used.custom.pct'] = (params.one - (double)(ctx['kubernetes.volume.fs.available.bytes']) / (double)(ctx['kubernetes.volume.fs.capacity.bytes'])) * params.to_percent",
          "params": {
            "to_percent": 100,
            "one": 1
          },
          "ignore_failure": true
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 7, 2023, 4:37pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/9 "2023-07-07T16:37:19Z")

</div>

Hmmm yes... I see that too --- head scratching I will take a look when I get a chance.  
Some form of syntax error

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 7, 2023, 4:57pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/10 "2023-07-07T16:57:55Z")

</div>

@patcan Got It!!! Wrong Syntax  
Remember I would use decimal params 1.0, 100.0

Not sure why the other syntax does not work but the shorthand seems to see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html#script-processor-access-source-fields)

> #### Access source fields
> 
> The script processor parses each incoming document’s JSON source fields into a set of maps, lists, and primitives. To access these fields with a Painless script, use the [map access operator](https://www.elastic.co/guide/en/elasticsearch/painless/8.8/painless-operators-reference.html#map-access-operator): `ctx['my-field']`. You can also use the shorthand `ctx.<my-field>` syntax.

```auto
PUT _ingest/pipeline/metrics-kubernetes.volume@custom
{
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": "ctx.temp_calc_result = (params.one - (double)(ctx.kubernetes.volume.fs.available.bytes) / (double)(ctx.kubernetes.volume.fs.capacity.bytes)) * params.to_percent",
        "params": {
          "to_percent": 100.0,
          "one": 1.0
        },
        "ignore_failure": true
      },
      "set": {
        "field": "kubernetes.volume.fs.used.custom.pct",
        "copy_from": "temp_calc_result"
      },
      "remove": {
        "field": "temp_calc_result"
      }
    }
  ]
}

# Simulate

POST _ingest/pipeline/metrics-kubernetes.volume@custom/_simulate
{
  "docs": [
    {
      "_index": ".test",
      "_id": "test",
      "_source": {
        "kubernetes": {
          "volume": {
            "name": "test",
            "fs": {
              "available": {
                "bytes": 262131712
              },
              "capacity": {
                "bytes": 262144000
              }
            }
          }
        }
      }
    }
  ]
}

# Results

{
  "docs": [
    {
      "doc": {
        "_index": ".test",
        "_id": "test",
        "_version": "-3",
        "_source": {
          "kubernetes": {
            "volume": {
              "name": "test",
              "fs": {
                "available": {
                  "bytes": 262131712
                },
                "used": {
                  "custom": {
                    "pct": 0.004687500000000178
                  }
                },
                "capacity": {
                  "bytes": 262144000
                }
              }
            }
          }
        },
        "_ingest": {
          "timestamp": "2023-07-07T16:56:24.869550011Z"
        }
      }
    }
  ]
}

# Now Test Real 

POST test-discuss/_doc/?pipeline=metrics-kubernetes.volume@custom
{
  "kubernetes": {
    "volume": {
      "name": "test",
      "fs": {
        "available": {
          "bytes": 262131712
        },
        "capacity": {
          "bytes": 262144000
        }
      }
    }
  }
}

# Results YAY!
GET test-discuss/_search
{
  "fields": [
    "*"
  ]
}

# Results
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "test-discuss",
        "_id": "mYpJMYkBuXkTucfoSgvb",
        "_score": 1,
        "_source": {
          "kubernetes": {
            "volume": {
              "name": "test",
              "fs": {
                "available": {
                  "bytes": 262131712
                },
                "used": {
                  "custom": {
                    "pct": 0.004687500000000178
                  }
                },
                "capacity": {
                  "bytes": 262144000
                }
              }
            }
          }
        },
        "fields": {
          "kubernetes.volume.fs.capacity.bytes": [
            262144000
          ],
          "kubernetes.volume.name": [
            "test"
          ],
          "kubernetes.volume.fs.available.bytes": [
            262131712
          ],
          "kubernetes.volume.name.keyword": [
            "test"
          ],
          "kubernetes.volume.fs.used.custom.pct": [
            0.0046875
          ]
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 7, 2023, 5:11pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/11 "2023-07-07T17:11:39Z")

</div>

Ahhhh I found it this would be the proper other syntax is a _ **map** _

`ctx['kubernetes']['volume']['fs']['available']['bytes']`

---

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 7, 2023, 9:06pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/12 "2023-07-07T21:06:36Z")

</div>

Yeap @stephenb - that's the solution. Thank you.

I had to add another set processor in front to set the new field.  
The script processor fails if the field doesn't exist,  
Full working JSON looks like this:

```auto
[
  {
    "set": {
      "field": "kubernetes.volume.fs.used.custom.pct",
      "value": "0",
      "override": false
    }
  },
  {
    "script": {
      "source": "ctx['kubernetes']['volume']['fs']['used']['custom']['pct'] = (params.one - (double)(ctx['kubernetes']['volume']['fs']['available']['bytes']) / (double)(ctx['kubernetes']['volume']['fs']['capacity']['bytes'])) * params.to_percent",
      "params": {
        "to_percent": 100,
        "one": 1
      },
      "ignore_failure": true
    }
  }
]

```

I just wonder how to round the result to 1 decimal, so let say max 10.5%

Should I use convert processor to integer or is there a better solution?

---

<div class="post-metadata">

**Author:** ![patcan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patcan/32/123002_2.png) [@patcan](https://discuss.elastic.co/u/patcan)\
**Post date:** [July 7, 2023, 9:15pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/13 "2023-07-07T21:15:07Z")

</div>

> [@patcan](#):
>
> `"ctx['kubernetes']['volume']['fs']['used']['custom']['pct'] = (params.one - (double)(ctx['kubernetes']['volume']['fs']['available']['bytes']) / (double)(ctx['kubernetes']['volume']['fs']['capacity']['bytes'])) * params.to_percent"`

Have it - solution is:

```auto
[
  {
    "set": {
      "field": "kubernetes.volume.fs.used.custom.pct",
      "value": "0",
      "override": false
    }
  },
  {
    "script": {
      "source": "ctx['kubernetes']['volume']['fs']['used']['custom']['pct'] = Math.round((params.one - (double)(ctx['kubernetes']['volume']['fs']['available']['bytes']) / (double)(ctx['kubernetes']['volume']['fs']['capacity']['bytes'])) * params.to_percent * 10.0) / 10.0",
      "params": {
        "to_percent": 100,
        "one": 1
      },
      "ignore_failure": true
    }
  }
]

```

Math.round

In the updated script, the result is multiplied by `10.0`, then rounded using `Math.round()`, and finally divided by `10.0` again to get the rounded value with one decimal place.

Note that the `Math.round()` function rounds the number to the nearest whole number, so multiplying and dividing by `10.0` allows you to round to one decimal place.

Thanks again

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2023, 9:15pm UTC](https://discuss.elastic.co/t/add-new-field-to-index-based-on-maths-calculation-from-other-fields-in-the-same-index/337571/14 "2023-08-04T21:15:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.