Remove field create by processor split target_field

Elastic Stack Elasticsearch
1

Here is my pipeline setting

{
    "processors": [
      {
        "split": {
          "field": "service",
          "separator": "/",
          "target_field": "test"
        }
      }
      {
        "script": {
          "source": """
           if (ctx.service.contains("serverid")) { /// for serverid
              ctx.metric_type = ctx.test[1]; 
             ctx.serverid = ctx.instancename;
            }
            else {
             ctx.metric_type = ctx.service;
            }
"""
        }
      },
      {
        "remove": {
          "field": [
            "test"
          ],
          "ignore_failure": true
        }
      }
    ]
}

I try to remove test field after whole process finished.
but it didn't.
Is there any way to do it?

2

hmm.. I tried your example and other then a missing , after split and before script, it seems to work.

What version of Elasticsearch ? Can you provide an example input ?

(copied your example, added the missing , and called the pipeline test)

POST _ingest/pipeline/test/_simulate
  {
  "docs": [
    {
      "_source": {
        "service" : "foo/bar/serverid/1/foo/baz",
        "instancename": "myinstance"
      }
    }
  ]
}

Results in :

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_type",
        "_id" : "_id",
        "_source" : {
          "instancename" : "myinstance",
          "service" : "foo/bar/serverid/1/foo/baz",
          "metric_type" : "bar",
          "serverid" : "myinstance"
        },
        "_ingest" : {
          "timestamp" : "2018-10-30T14:12:03.38385Z"
        }
      }
    }
  ]
}
3

My ES version is 6.4.2
Here is my simulator

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "test",
    "processors": [
      {
        "split": {
          "field": "service",
          "separator": "/",
          "target_field": "test"
        }
      },
      {
        "gsub": {
          "field": "service",
          "pattern": "_ld",
          "replacement": ""
        }
      },
      {
        "script": {
          "source": """
            if (ctx.service.contains("routerid")) { /// for routerid
             ctx.metric_type = ctx.test[1];
             ctx.routerid = ctx.instancename;
            } else {
             ctx.metric_type = ctx.service;
            }
"""
        }
      },
      {
        "remove": {
          "field": [
            "instancename",
            "test"
          ],
          "ignore_failure": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "my-index",
      "_type": "my-type",
      "_id": "my-id",
      "_source": {
        "service": "routerid/connected_players",
        "instancename": 2141007
      }
    }
  ]
}

And result

{
  "docs": [
    {
      "doc": {
        "_index": "my-index",
        "_type": "my-type",
        "_id": "my-id",
        "_source": {
          "test": [
            "routerid",
            "connected_players"
          ],
          "service": "routerid/connected_players",
          "routerid": 2141007,
          "metric_type": "connected_players"
        },
        "_ingest": {
          "timestamp": "2018-10-31T01:24:44.560Z"
        }
      }
    }
  ]
}

The test field still exist.

4

This is really odd.. I ran the exact simulate in and got the following:

{
  "docs": [
    {
      "doc": {
        "_index": "my-index",
        "_type": "my-type",
        "_id": "my-id",
        "_source": {
          "metric_type": "connected_players",
          "service": "routerid/connected_players",
          "routerid": 2141007
        },
        "_ingest": {
          "timestamp": "2018-10-31T14:42:17.690913Z"
        }
      }
    }
  ]
}

To ensure I didn't have anything odd locally, i ran this from the following docker compose

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.4.2
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - node.ingest=false
      - node.name=es1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
    networks:
      - esnet
  kibana:
    image: docker.elastic.co/kibana/kibana:6.4.2
    ports:
      - "5601:5601"
    networks:
      - esnet
    depends_on:
      - elasticsearch

networks:
  esnet:

Can you post the result of:

GET /?filter_path=version

and the result of ?verbose simulation

POST _ingest/pipeline/_simulate?verbose
... your example above ...
5 
GET /?filter_path=version

{
  "version": {
    "number": "6.4.2",
    "build_flavor": "default",
    "build_type": "rpm",
    "build_hash": "04711c2",
    "build_date": "2018-09-26T13:34:09.098244Z",
    "build_snapshot": false,
    "lucene_version": "7.4.0",
    "minimum_wire_compatibility_version": "5.6.0",
    "minimum_index_compatibility_version": "5.0.0"
  }
}

It is very odd. Because it works now....
And I already done the reindex, the test field still exist.
Anyway seems fine now. I will try that again

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.