Removing a field with dots in an ingest pipeline

[Anabella_Cristaldi (Anna)]

Hi all,
I'm using filebeat with an ingest simple ingest pipeline and I want to remove some of the host's field, for example the host.os

I've defined a ingest pipeline

PUT _ingest/pipeline/appsalutmw
{
  "description": "APP Salut Ingest Pipeline",
  "processors": [
    {
      "set": {
        "field": "event.module",
        "value": "appsalutmw"
      }
    },
    {
      "remove": {
        "field": "host.os.name"
      }
    }
  ]
}

and when I simulate (using the same json I receive from filebeat) I get the error:

 {
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "illegal_argument_exception",
            "reason" : "field [host] not present as part of path [host.os.name]"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "field [host] not present as part of path [host.os.name]"
      }
    }
  ]
}

I've tried to used (before the remove processor) the dot_expander but with not luck

POST _ingest/pipeline/appsalutmw/_simulate
{
  "docs": [
    {
      "_source": {
        "host.os.name": [
          "Red Hat Enterprise Linux Server"
        ]
      }
    }
  ]
}

I'm able to remove it with an script processor but I was wondering why I'm not able to do it with the remove processor

Thank you
Regards
Ana

[stephenb (Stephen Brown)]

Hi @Anabella_Cristaldi

In that second example can you show the entire simulate plus the dock doc or at least the host.os.name portion so we can see.

It does seem like the dot expander issue but I know you said you tried it.

Show me the whole simulate pipeline plus the document plus the error.

[stephenb (Stephen Brown)]

Interesting... the dot_expander works a little difference that I expected ..

PUT _ingest/pipeline/test-discuss
{
  "description": "APP Salut Ingest Pipeline",
  "processors": [
    {
      "dot_expander": {
        "field": "host.os.name"
      }
    },
    {
      "dot_expander": {
        "field": "host.os.rev"
      }
    },
    {
      "set": {
        "field": "event.module",
        "value": "appsalutmw"
      }
    },
    {
      "remove": {
        "field": "host.os.name",
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "field": "my.test.field",
        "ignore_failure": true
      }
    }
  ]
}

Then simulate

POST /test-discuss/_doc?pipeline=test-discuss
{
  "host.os.name": "Red Hat Enterprise Linux Server",
  "host.os.rev": "32",
  "my": {
    "test": {
      "field": "value",
      "other_field" : "other_value"
    }
  }
}

Then results

GET /test-discuss/_search

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-discuss",
        "_type" : "_doc",
        "_id" : "rVeWWXkBl0GsIo33gNji",
        "_score" : 1.0,
        "_source" : {
          "host" : {
            "os" : {
              "rev" : "32"
            }
          },
          "my" : {
            "test" : {
              "other_field" : "other_value"
            }
          },
          "event" : {
            "module" : "appsalutmw"
          }
        }
      }
    ]
  }
}

[Anabella_Cristaldi (Anna)]

Thank you