How to merge parent and child documents in Elasticsearch into a single enriched record?

Elastic Stack Kibana
1

Hi everyone,

I’m working with an Elasticsearch index that stores HTTP request logs. Each request is represented by two documents:

  • A parent document with a unique ID field.
  • A child document that contains a parentID field referencing the parent’s ID.

Both documents share the same structure (i.e., same fields), but with different values. I’d like to merge each parent-child pair into a single document, either in the same index or a new one. Ideally, the merged document would:

  • Retain all fields from the parent as-is.
  • Include all fields from the child, but with a child_ prefix (e.g., child_status, child_timestamp, etc.).

What’s the best way to achieve this in Elasticsearch/Kibana?

  • Should I use a transform job, ingest pipeline, or script with _update_by_query?
  • Is there a way to do this efficiently for large datasets?

Thanks in advance for your help!

2

anyone ?

3

Hello @verza

Welcome to the community.

Looking at your usecase as per my understanding for large datasets you should go for transform job on common field id & can include the fields as per the requirement.

Thanks!!

4

Can i still do it if the common field id are different ? In the parent the field is named 'ID' and in the Child is named 'ParentID'

5

Hello @verza

In this scenario we might have to use an enrich policy by which data for 1 index is appended to index 2 based on ID field.

Thanks!!