Seperate the elements into multiple documents using ingest pipeline

Elastic Stack Elasticsearch
1

Hi

I want to create multiple documents/events based on the target field. Target field contains multiple elements and want to seperate it and create multiple documents using ingest pipeline. I tried with foreach and script processor but it is not working. We can't use logstash here as there are some restrictions.

Data looks like this,

{"ErrorCode":"E00065427","errorSummary":"Sorry, there's been an error.","errorId":"hshehje-yehhbs","target":[{"id":"5266262727","type":"A","display_name","X","alternate_id":"unknown"},{"id":"526727278327","type":"B","display_name":"Y","alternate_id":"unknown"},{"id":"6636377373","type":"C","display_name":"Z","alternate_id":"unknown"}]}

2

Ingest pipelines can not split a document into multiple ones, only Logstash can.

3

How are you indexing the data?

This is something that is not possible to do with Ingest Pipelines.

It can be easily done with Logstash using the split filter and some inputs of Elastic Agent can also do the same split, basically the Kafka input and the httpjson have ways to split data into multiple events before sending the data to Elasticsearch.

1 Like
4

This is a frequently-requested processor that is unfortunately much more difficult to implement than you would think. Here is a summary: [Ingest Pipeline] Ability to split documents · Issue #56769 · elastic/elasticsearch · GitHub