Regarding Ingest Node Set Processor

[dawiro]

Hi,
I'm looking at upgrading from 5.6.3 to 6.8.x and have come across a behaviour in an ingest node pipeline which I need to address before upgrading...

I have the following processor defined in one of our pipelines:

{
        "set" : {
          "field" : "pipeline",
          "value" : "metadata"
        }
}

In our 5.6.3 cluster this ends up being indexed as:

        "pipeline" : {
            "properties" : {
              "name" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
          }
        }

While in another 6.8.0 cluster:

          "pipeline" : {
            "type" : "keyword",
            "ignore_above" : 256
          }

This will lead to problems on upgrade. I have seen that logstash 6.8.x will fail to write to this pipeline if the index contains the 5.6.3 mapping shown above. Our current logstashes are at 5.6.9.

Why is this happening and how do you recommend I handle it? Remove that processor until the upgrade is complete?

Regards,
D

[spinscale (Alexander Reelsen)]

Hey,

this sounds less like an ingest node processor issue (as the ingest node only changes the JSON document), but more like a mapping issue. Are you missing or using a different index template on the different elasticsearch versions?

--Alex

[dawiro]

Figured it out this morning. Someone is shipping rogue fields to that cluster outside of the pipeline..