Ingest pipeline, "remove" nested fields

[Russell_Fulton]

Version 7.14

I am attempting to filter a winlogbeats stream in an ingest pipeline. One thing I want to do is strip out the whole agent tree as this is repeated in every record.

Is there a way to remove "agent.*" in one go?

Aside: it is also unclear to me if these field are "flattened" (i.e. do I need to use dot expander) and how would I know.

[stephenb (Stephen Brown)]

Technically I believe the agent fields has sub-objects and is not nested.
See here

You do not need the dotexpander and you can just name the top field in your case agent if you completely want to remove the whole agent tree

PUT /_ingest/pipeline/test-remove
{
  "processors" : [
    {
      "remove": {
        "field": "field-b"
      }
    }
  ]
}


# This is how you simulated a pipeline
POST /_ingest/pipeline/test-remove/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "field-a": "1234",
        "field-b":
        {
          "sub1" : "value1",
          "sub2" : "value2"
        }
      }
    }
  ]
}

results note entire field-b is removed

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "field-a" : "1234"
        },
        "_ingest" : {
          "timestamp" : "2021-09-12T20:54:48.4217817Z"
        }
      }
    }
  ]
}

[Russell_Fulton]

Thanks! that is exactly what I needed.
works as advertised : )