How can you write into 2 different indexes using a ingest pipeline?
Example. a single doc with fields a , b, c, and d ?
index A will receive fields A & B
index B will receive fields C & D
How can I accomplish this with ingest pipelines?
You can not as far as I know. You need to do that split before you send the documents to Elasticsearch.
Seconding what @Christian_Dahlqvist said. An ingest pipeline can write to only one index. So you can't split a document in two documents within an ingest pipeline.
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
There maybe some options, with logstash for example
Ingest with logstash use the clone plugin to duplicate the document, tag (A,B: the original A and clone B), then use logic to send those tagged A to pipeline A, those tagged B to pipeline B. Then use the prune plugin to remove unwanted fields.
There is an example of this in this blog post: Using Logstash to Split Data and Send it to Multiple Outputs | Elastic Blog.
As for ingest you could likely do a pipeline to pipeline
Alternatively, use logstash with 'pipeline to pipeline' Pipeline-to-Pipeline Communication | Logstash Reference [7.13] | Elastic
For an ingest pipeline, use reindex from the source index to another index (need _source enabled on docs to use reindex) Ingest pipelines | Elasticsearch Guide [7.13] | Elastic