Ingest pipeline, "remove" nested fields

Elastic Stack Elasticsearch
1

Version 7.14

I am attempting to filter a winlogbeats stream in an ingest pipeline. One thing I want to do is strip out the whole agent tree as this is repeated in every record.

Is there a way to remove "agent.*" in one go?

Aside: it is also unclear to me if these field are "flattened" (i.e. do I need to use dot expander) and how would I know.

2

Technically I believe the agent fields has sub-objects and is not nested.
See here

You do not need the dotexpander and you can just name the top field in your case agent if you completely want to remove the whole agent tree

PUT /_ingest/pipeline/test-remove
{
  "processors" : [
    {
      "remove": {
        "field": "field-b"
      }
    }
  ]
}


# This is how you simulated a pipeline
POST /_ingest/pipeline/test-remove/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "field-a": "1234",
        "field-b":
        {
          "sub1" : "value1",
          "sub2" : "value2"
        }
      }
    }
  ]
}

results note entire field-b is removed

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "field-a" : "1234"
        },
        "_ingest" : {
          "timestamp" : "2021-09-12T20:54:48.4217817Z"
        }
      }
    }
  ]
}
3

Thanks! that is exactly what I needed.
works as advertised : )

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.