Hi!
I have a 6 node cluster (ES 6.2.3) with a very basic ingestion pipeline. When issuing a
...saveToEs("...", Map("es.mapping.id"->"...", "es.nodes"->"...", "es.ingest.pipeline"->"parse_event"))
some of the Spark-executors fail with the following error: "pipeline with id [parse_event] does not exist", and it looks like none of the rows are commited to ES for that executor. However, other executors running in parallell completes successfully. The pipeline exists and all of the nodes are "ingest nodes". The error seems to appear randomly and the failed executor gets re-scheduled BUT it is very unclear what's going on in the cluster. Looking in the log files doesn't help either as the error above is all that's reported in those files as well.
From what I've seen there seems to be some strange pipeline behavior being reported earlier... Does anybody have a remedy for this?
Many thanks in advance!