How can you write into 2 different indexes using a ingest pipeline?
Example. a single doc with fields a , b, c, and d ?
index A will receive fields A & B
index B will receive fields C & D
How can I accomplish this with ingest pipelines?
How can you write into 2 different indexes using a ingest pipeline?
Example. a single doc with fields a , b, c, and d ?
index A will receive fields A & B
index B will receive fields C & D
How can I accomplish this with ingest pipelines?
You can not as far as I know. You need to do that split before you send the documents to Elasticsearch.
Seconding what @Christian_Dahlqvist said. An ingest pipeline can write to only one index. So you can't split a document in two documents within an ingest pipeline.
There maybe some options, with logstash for example
Ingest with logstash use the clone plugin to duplicate the document, tag (A,B: the original A and clone B), then use logic to send those tagged A to pipeline A, those tagged B to pipeline B. Then use the prune plugin to remove unwanted fields.
There is an example of this in this blog post: Using Logstash to Split Data and Send it to Multiple Outputs | Elastic Blog.
As for ingest you could likely do a pipeline to pipeline
Alternatively, use logstash with 'pipeline to pipeline' Pipeline-to-Pipeline Communication | Logstash Reference [7.13] | Elastic
For an ingest pipeline, use reindex from the source index to another index (need _source enabled on docs to use reindex) Ingest pipelines | Elasticsearch Guide [7.13] | Elastic
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.