# Elasticsearch job to end messages with beginning messages, merge, then reindex **URL:** \ **Category:** Elasticsearch\ **Created:** [November 24, 2020, 4:44pm UTC](https://discuss.elastic.co/t/elasticsearch-job-to-end-messages-with-beginning-messages-merge-then-reindex/256541 "2020-11-24T16:44:16Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![aerodynamic](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@aerodynamic](https://discuss.elastic.co/u/aerodynamic)\ **Post date:** [November 24, 2020, 4:44pm UTC](https://discuss.elastic.co/t/elasticsearch-job-to-end-messages-with-beginning-messages-merge-then-reindex/256541/1 "2020-11-24T16:44:16Z")
The system I'm running is currently matching beginning messages with end messages then sending them to logstash for further processing. This is done through a web of python code. I was wondering if this could be accomplished with native elastic. Any suggestions? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 25, 2020, 12:37am UTC](https://discuss.elastic.co/t/elasticsearch-job-to-end-messages-with-beginning-messages-merge-then-reindex/256541/2 "2020-11-25T00:37:46Z")
It could be done with a multiline filter in Filebeat or Logstash to merge them. ---
**Author:** ![aerodynamic](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@aerodynamic](https://discuss.elastic.co/u/aerodynamic)\ **Post date:** [November 25, 2020, 2:10pm UTC](https://discuss.elastic.co/t/elasticsearch-job-to-end-messages-with-beginning-messages-merge-then-reindex/256541/3 "2020-11-25T14:10:23Z")
So there's a catch that the messages don't show up at the same time. Would it be possible to sit on to-be-matched documents in FB/LS until the complimentary message arrives? ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [November 25, 2020, 3:44pm UTC](https://discuss.elastic.co/t/elasticsearch-job-to-end-messages-with-beginning-messages-merge-then-reindex/256541/4 "2020-11-25T15:44:21Z")
Keeping messages until another message of the same kind arrives has its limitations: - resources for keeping the messages, - in memory is not failsafe - what if the corresponding end message never arrives? TTL? - messages must be routed to the same worker in order to match, this might introduce hot spot workers You have to decide whether you are ok with such limitations. An alternative to that is indexing the messages and building sessions out of it in a 2nd pass. Such a system can be realized with [Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-overview.html). The transform can be setup as continuous transform, so it would session'ize your data as it comes in. This approach is more scalable and fail safe, however there is no free lunch: You require more resources, e.g. you have a larger source index. The source index however could be managed by ILM, if you only care about the sessions you could use a rolling index that gets deleted e.g. every week. We have an example for calculating the [duration between 2 events](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-bucket-script), I think this is potentially useful for you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 23, 2020, 3:44pm UTC](https://discuss.elastic.co/t/elasticsearch-job-to-end-messages-with-beginning-messages-merge-then-reindex/256541/5 "2020-12-23T15:44:36Z")
Closed on December 23, 2020, 3:44pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.