# Updating a field in a doc with the fields been obtained from another dataset (basically joins like feature)

**URL:** <https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080>\
**Category:** Elasticsearch\
**Created:** [April 3, 2019, 12:32am UTC](https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080 "2019-04-03T00:32:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)\
**Post date:** [April 3, 2019, 12:32am UTC](https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080/1 "2019-04-03T00:32:21Z")

</div>

I have an index with data as :  
{  
"\_index" : "qstat-2019.03.29",  
"\_type" : "doc",  
"\_source" : {  
"Job\_Number" : "150153525",  
"Job\_State" : "pending",  
"hard\_req\_queue" : "all.q",  
"Submission\_time" :  
}

As you could see that "Submission\_time" field is empty which I would get from another dataset and was getting that data set into elastic into a different index and trying to create join between both the indexes based on "Job\_Number" field. But as joins are not supported so is there a way that I can stream in the second data source and update the value of "Submission\_time" in above shown index by matching the "job\_Number" field.

My both the data sources are xml files so I am parsing the xml file in logstash before writing the data into elastic. So, I would parse the second xml in logstash to extract "job\_Number" and "Submission\_time" and write the later to the index doc shown above based on the matching "job\_Number"

---

<div class="post-metadata">

**Author:** ![sverma](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sverma](https://discuss.elastic.co/u/sverma)\
**Post date:** [April 10, 2019, 2:13am UTC](https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080/2 "2019-04-10T02:13:05Z")

</div>

Any help on this please

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 10, 2019, 10:50am UTC](https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080/3 "2019-04-10T10:50:12Z")

</div>

I'd use logstash to read from elasticsearch, add an elasticsearch filter to do lookups and an elasticsearch output to write to elasticsearch the result.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [April 10, 2019, 11:00am UTC](https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080/4 "2019-04-10T11:00:38Z")

</div>

Maybe you should think about maintaining an [entity-centric index](https://twitter.com/elasticmark/status/1009380268409610240) built from these events?

In your case the entity would be a "job". The example document looks like a state-change event and several of these could be summarised in a job entity. The advantage of this approach is:

1. You can reduce the cost of joining fast-changing data (multiple events can be batched into a single update to the job entity)
2. Custom attributes can be derived from multiple events e.g. "jobDuration" or "lastStatus".
3. Your update scripts can see _all_ job history and tag anomalies eg where state changes don't follow expected sequences

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2019, 11:00am UTC](https://discuss.elastic.co/t/updating-a-field-in-a-doc-with-the-fields-been-obtained-from-another-dataset-basically-joins-like-feature/175080/5 "2019-05-08T11:00:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.