# Ingest pipeline, rename a field that has a dynamic part

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488>\
**Category:** Elasticsearch\
**Created:** [January 17, 2020, 3:47pm UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488 "2020-01-17T15:47:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wedkarz014](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wedkarz014/32/48935_2.png) [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\
**Post date:** [January 17, 2020, 3:47pm UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488/1 "2020-01-17T15:47:33Z")

</div>

hi,  
I have a metrics with fields which look like that:

> ```
> prometheus.xyz-dev.label.name
> prometheus.xyz-dev.tomcat_global_error_total.value
> prometheus.xyz-dev.tomcat_global_received_bytes_total.value
> prometheus.xyz-dev.tomcat_global_request_max_seconds.value
> prometheus.xyz-dev.tomcat_global_request_seconds.count
> 
> ```

I would like to delete first two parts, but the second one is a kubernetes.namespace value so this is a dynamic value.  
This is what i need:  
prometheus.xyz-dev.label.name =\> label.name  
prometheus.abc-prod.label.name =\> label.name

or eventually  
prometheus.xyz-dev.label.name =\> prometheus.label.name

I need to create an ingest pipeline which can to that, but i have no idea how to do that. Can you help?

---

<div class="post-metadata">

**Author:** ![wedkarz014](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wedkarz014/32/48935_2.png) [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\
**Post date:** [January 22, 2020, 5:00pm UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488/2 "2020-01-22T17:00:43Z")

</div>

Ok, I partly resolve that problem, I created two pipelines, the first one checks metricset.module and runs another pipeline which changes fields name. But I have to define each field name (right now its more than 60), this is not a generic solution. Do you have any idea how can i do that?

first pipeline:

> ```
> {
> "set" : {
> "field" : "metricset_module",
> "value" : "{{metricset.module}}",
> "ignore_failure" : true
> }
> },
> {
> "pipeline" : {
> "name" : "prometheus_pipeline",
> "if" : "ctx.containsKey('metricset_module')&&ctx.metricset_module!=''"
> }
> },
> {
> "remove" : {
> "field" : [
> "metricset_module"
> ],
> "ignore_failure" : true
> }
> }
> 
> ```

and

> ```
> "prometheus_pipeline" : {
> "description" : "",
> "processors" : [
> {
> "rename" : {
> "if" : "ctx.metricset.module=='prometheus'",
> "field" : "prometheus.{{kubernetes.namespace}}.http_server_requests_seconds.count",
> "target_field" : "prometheus.http_server_requests_seconds.count",
> "ignore_failure" : true
> }
> },
> {
> "rename" : {
> "if" : "ctx.metricset.module=='prometheus'",
> "field" : "prometheus.{{kubernetes.namespace}}.http_server_requests_seconds.sum",
> "target_field" : "prometheus.http_server_requests_seconds.sum",
> "ignore_failure" : true
> }
> },
> {
> "rename" : {
> "if" : "ctx.metricset.module=='prometheus'",
> "field" : "prometheus.{{kubernetes.namespace}}.http_server_requests_seconds_max.value",
> "target_field" : "prometheus.http_server_requests_seconds_max.value",
> "ignore_failure" : true
> }
> },
> ...etc
> 
> ```

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 23, 2020, 11:09am UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488/3 "2020-01-23T11:09:20Z")

</div>

Hi there!

Are the first two parts (`prometheus.xyz-dev` and `prometheus.abc-prod`) always the same?

I mean, are they static and equivalent to a limited set of values or can they change randomly?

Also, please paste here a sample of input documents and tell us if by any chance you're using logstash in your stack to ingest data or if you're ingesting them directly into elastic from another source.

Thanks

---

<div class="post-metadata">

**Author:** ![wedkarz014](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wedkarz014/32/48935_2.png) [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\
**Post date:** [January 24, 2020, 3:46pm UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488/4 "2020-01-24T15:46:20Z")

</div>

First part so "prometheus" is static, but the next one not. This is the name of namespace from kubernetes, so we have a lot of namespaces and we will have a lot of new. And then there is a name of metric. As you can see in my second message, i did it (delete second part - namespace) but i had to write rename pipeline for each metricname. There are currently about 60 metric names, but there may be more in the future. So i would like to do sth dynamic  
We skip the logstash, flow is from metricbeat to elasticsearch

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 24, 2020, 4:37pm UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488/5 "2020-01-24T16:37:55Z")

</div>

Ok so, if the first part is always `prometheus` then you can do something like:

```
POST _ingest/pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "old_field": "prometheus.whatever.my_app"
      }
    }
  ]
  , "pipeline": {
    "processors": [
      {
        "script": {
          "source": """
            if (ctx.containsKey('old_field') && ctx.old_field instanceof String) {
              def new_field_start = 12 + ctx.old_field.substring(11).indexOf('.');
              ctx.new_field = ctx.old_field.substring(new_field_start);
            }
          """
        }
      }
    ]
  }
}

```

It returns:

```
{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "new_field" : "my_app",
          "old_field" : "prometheus.whatever.my_app"
        },
        "_ingest" : {
          "timestamp" : "2020-01-24T16:36:31.987Z"
        }
      }
    }
  ]
}

```

It is not very nice to see but it should fit your case. I don't remember how (and if you can) to find the second occurrence of a string in painless without using the regex (to use them you have to enable them in the elasticsearch.yml with the consequent elasticsearch warning about it).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2020, 4:37pm UTC](https://discuss.elastic.co/t/ingest-pipeline-rename-a-field-that-has-a-dynamic-part/215488/6 "2020-02-21T16:37:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.