# Bidirectional asynchronous data enrichment

**URL:** <https://discuss.elastic.co/t/bidirectional-asynchronous-data-enrichment/212233>\
**Category:** Elasticsearch\
**Created:** [December 17, 2019, 9:41pm UTC](https://discuss.elastic.co/t/bidirectional-asynchronous-data-enrichment/212233 "2019-12-17T21:41:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ea1987](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ea1987/32/51356_2.png) [@ea1987](https://discuss.elastic.co/u/ea1987)\
**Post date:** [December 17, 2019, 9:41pm UTC](https://discuss.elastic.co/t/bidirectional-asynchronous-data-enrichment/212233/1 "2019-12-17T21:41:20Z")

</div>

Hi guys,  
I'm currently performing data enrichment of 2 different type of events (type A and type B) which are ingested and stored in different times in Elasticsearch.  
Keep in mind that:

1. Events are correlated by a correlation\_id field
2. Correlation ratio typeA/typeB is: 1 to n
3. Events are written in different times, event type A first
4. Some fields of event type A need to be written in events type B and viceversa

Following enrichment pattern and scenario details:

**ENRICHMENT PATTERN SUMMARY**  
During events type A and B ingestion I basically write/update an event (\_id=correlation\_id) on a specific enrichment index, with desired enrichment data of both events. Later, with a scheduled pipeline, I enrich type A/B events using values of the enrichment event.

**2 DIFFERENT EVENTS**

```
Event type A:
- field1 (correlation_id)
- field2
- field3
- field7

Event type B:
- field1 (correlation_id)
- field4
- field5
- field6

```

**3 INDICES**

```
INDEX 1: storing event of type A
INDEX 2: storing event of type B
INDEX 3: storing fields of both events (field2, field4, field5, field7)

```

**DATA INGESTION/ENRICHMENT STEPS**

```
t0 -> Ingestion of Event type A to INDEX 1, ingestion of an event containing field1,field2,field7 to INDEX 3 using field1 value as _id
t1 -> Ingestion of Event type A to INDEX 2, update of event stored in INDEX 3, adding field4, field5 to it
t2 -> Scheduled enrichment (update) of Event type A and Event type B using data of INDEX 3 (queried per field1)

```

PRO of this solution:

- no additional load during data ingestion on t0 and t1

According to your experience:

1. Is this the best asynchronous enrichment solution?
2. Is this something that could be done using the new enrich processor feature of ES 7.5? It doesn't seem so because in this scenaria data need to be enriched in both ways (from index 1 to 2 and viceversa)
3. do you have any advices?

Thank you in advance,

Andrea

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 14, 2020, 9:41pm UTC](https://discuss.elastic.co/t/bidirectional-asynchronous-data-enrichment/212233/2 "2020-01-14T21:41:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.