Is reindex'ing to a pipeline broken between 6.0 & 6.1?

Elastic Stack Elasticsearch
1

We may be overlooking something here, but are running out of avenues of investigation.

In 6.0 we could reindex to a pipeline whereby the pipeline dynamically routed the docs to a number of different indexes. The definition of the index in the reindex task had to exist but was useless so we had the literal one from the documentation. { "index" : "dest" } and it was just ignored.

Since upgrading to 6.1.1 it appears that the reindex job creates a new index called "dest" and the pipeline is no longer used.

Is this expected behaviour in 6.1.1 ?

2

A bit more info. The subset of data we were reindexing was all failing on the pipeline due to mappings. We knew this and were ok with it. Previously this would appear in the failure array of the reindex task.

Now however pipeline failures appear to be going into this dummy index and there are no records in the reindex tasks failure array. Meanwhile docs from a subset of the data that do successfully get through the pipeline I now realise are correctly assigned to different indices as required. I can't recall if this was happening on V6.0.

So since we MUST include an index in the reindex task it looks like its no longer possible to see docs that failed to get through the pipeline from the perspective of the reindex task. This was needed as it detailed the failure message.

Is THIS expected behaviour ?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.