# Entity-centric indexing with Transforms

**URL:** <https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [July 5, 2021, 9:15am UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798 "2021-07-05T09:15:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![katja1](https://avatars.discourse-cdn.com/v4/letter/k/b487fb/32.png) [@katja1](https://discuss.elastic.co/u/katja1)\
**Post date:** [July 5, 2021, 9:15am UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798/1 "2021-07-05T09:15:57Z")

</div>

Hello, we're working with Elasticsearch for the first time and we are currently deciding on what would be the best solution for our problem at hand.

We are receiving Event based logs (in JSON form) from our applications directly to Elasticsearch index. These logs are highly interconnected (they share a common unique ID) and therefore we need to convert/aggregate them in an Entity-centered fashion.

Each event usually has a status change in the target field. There are more statuses than just start/end. Document has more data which can be used to create more than just one Entity-centered index.

```auto
{
*uniqueID*: ain123in145512kn
name: Bob
target: {
eventStart: {timestamp: 2020-06-01T13:50:55.000Z}
}
}

```

```auto
{
*uniqueID*: ain123in145512kn
name: Bob
target: {
eventStop: {timestamp: 2021-06-01T13:50:55.000Z}
}
}

```

We were already able to join these documents using Python or Logstash. We basically created an index that contains the following documents:

```auto
{
*uniqueID*: ain123in145512kn
name: Bob
target: {
eventStart: {timestamp: 2020-06-01T13:50:55.000Z},
eventStop: {timestamp: 2021-06-01T13:50:55.000Z}
*time_dif_Start_Stop : xxxx*
}
}

```

We assigned all events document ID that is the same as uniqueID which updated them automatically. Next step just calculated the difference between eventStart and eventStop timestamps.

We have certain requirements for our pipeline so we would prefer if data never has to leave elasticsearch. Therefore, we are wondering **if it is possible to do this with any of the tools that already exist in the ELK stack or are hosted in the Elastic cloud?** We tried using Transforms but we were only able to calculate aggregated fields in a new index. Is it possible to also basically merge/update all the documents into a single one with this tool or any other? It would be ideal for us as it is running on a schedule and we do not need any external tools to modify documents.

Any other suggestions or help would also be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![przemekwitek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/przemekwitek/32/79526_2.png) [@przemekwitek](https://discuss.elastic.co/u/przemekwitek)\
**Post date:** [July 5, 2021, 1:36pm UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798/2 "2021-07-05T13:36:14Z")

</div>

Hi,

It seems like transforms should be fitting your needs but it would be good to know more details.

> [@katja1](#):
>
> We tried using Transforms but we were only able to calculate aggregated fields in a new index.

If you tried using transforms, could you show the config you were using for that?  
What do you mean by "only able to calculate aggregated fields"? `eventStart` should be a result of `min` aggregation. Similarly `eventStop` should be a result of `max` aggregation.  
Is it `time_dif_Start_Stop` that is problematic for you? It looks like it could be calculated an ingest pipeline attached to your destination (entity-centric) index.

---

<div class="post-metadata">

**Author:** ![katja1](https://avatars.discourse-cdn.com/v4/letter/k/b487fb/32.png) [@katja1](https://discuss.elastic.co/u/katja1)\
**Post date:** [July 6, 2021, 10:03am UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798/3 "2021-07-06T10:03:44Z")

</div>

No, the `time_dif_Start_Stop` is not problematic, we are able to caluculate it with scripted metrics, and write it to destination index. What we are wondering is how to also "transfer" some of the existing fields (that are not part of aggregations and calcualtions) from the source index to the destination index, based on the shared ID (`uniqueID`)

---

<div class="post-metadata">

**Author:** ![przemekwitek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/przemekwitek/32/79526_2.png) [@przemekwitek](https://discuss.elastic.co/u/przemekwitek)\
**Post date:** [July 6, 2021, 12:22pm UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798/4 "2021-07-06T12:22:29Z")

</div>

If there are not many such fields, you can put them in the `group_by` section of the transform config.  
Of course, in such case they are not meant to be used for grouping (as grouping is achieved by having `uniqueID`) but they will be present in the destination index.

Please note, however, that if you have many such fields, it can impact performance of the transform.

---

<div class="post-metadata">

**Author:** ![katja1](https://avatars.discourse-cdn.com/v4/letter/k/b487fb/32.png) [@katja1](https://discuss.elastic.co/u/katja1)\
**Post date:** [July 7, 2021, 12:34pm UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798/5 "2021-07-07T12:34:51Z")

</div>

Thanks for your answer, that is what we needed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 4, 2021, 12:35pm UTC](https://discuss.elastic.co/t/entity-centric-indexing-with-transforms/277798/6 "2021-08-04T12:35:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.