Pipeline remove processor not removing all fields

Elastic Stack Elasticsearch
1

I have an ingest pipeline to remove some fields for a reindex. However, the pipeline is not removing the last field. I have changed the order and it always leaves the last field in line. I am Version: 6.4.2

{
  "reindex-prediction-clean": {
    "description": "Redindex prediction to clean fields",
    "processors": [
      {
        "remove": {
          "field": [
            "message_ovo",
            "event_from",
            "ovoid",
            "ovosev",
            "ovostate",
            "ticket_number",
            "sanity_check",
            "ovoeventtime",
            "ovo_event_time",
            "elapsed_time_sec"
          ],
          "ignore_failure": true
        }
      }
    ]
  }
}

Capture
Capture615×886 36.7 KB

2

@chapmantrain not a permanent fix but have you tried this to see what it does?

{
  "reindex-prediction-clean": {
    "description": "Redindex prediction to clean fields",
    "processors": [
      {
        "remove": {
          "field": [
            "message_ovo",
            "event_from",
            "ovoid",
            "ovosev",
            "ovostate",
            "ticket_number",
            "sanity_check",
            "ovoeventtime",
            "ovo_event_time",
            "elapsed_time_sec",
            ""
          ],
          "ignore_failure": true
        }
      }
    ]
  }
}
3

Hey Aaron, thanks for the quick reply. I gave it a spin but elastic just decided it would not remove any fields. Good idea though.

Norm

4

I think I got it. Probably should have RTFM more closely. Instead of ignore_failure I should have had ignore_missing: true. Looks like the pipline stopped processing when it hit a missing or null field in the parent.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.