# Use logstash to transform a message

**URL:** <https://discuss.elastic.co/t/use-logstash-to-transform-a-message/231811>\
**Category:** Logstash\
**Created:** [May 8, 2020, 11:25pm UTC](https://discuss.elastic.co/t/use-logstash-to-transform-a-message/231811 "2020-05-08T23:25:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dooger21](https://avatars.discourse-cdn.com/v4/letter/d/b4bc9f/32.png) [@dooger21](https://discuss.elastic.co/u/dooger21)\
**Post date:** [May 8, 2020, 11:25pm UTC](https://discuss.elastic.co/t/use-logstash-to-transform-a-message/231811/1 "2020-05-08T23:25:02Z")

</div>

```auto
Hello everyone,

I have a log that generates thousands of events a day, a group of events and its entire plot can be differentiated and obtained by the KEY field generated by the application as in the following example.

DATE | EVENT | KEY |QUEUE	
---------------------------------------------------------------------------------------------------------------
2020-04-09T10:54:54 EveIni 202004091554541340000000 App1
2020-04-09T11:48:10 EveGet 202004091554541340000000 Verify
2020-04-09T11:48:14 EveCon 202004091554541340000000 Verify
2020-04-09T11:58:21 EveGet 202004091554541340000000 UserVerify
2020-04-09T11:58:21 EveCon 202004091554541340000000 UserVerify
2020-04-09T13:58:22 EveGet 202004091554541340000000 App2
2020-04-09T13:58:22 EveCon 202004091554541340000000 App2
2020-04-09T13:58:22 EveEnd 202004091554541340000000 Process

What is required is for a KEY to group all its messages (EVENT + QUEUE) into a single elasticsearch document as below, using Logstash.

Date |Key |Msj1 |Msj2 |Msj3 |Msj4 |Msj5 |Msj6 |Msj7 |Msj8						
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-04-09T13:58:22 202004091554541340000000	EveIni-App1 EveGet-Verify	EveCon-Verify	EveGet-UserVerify	EveCon-UserVerify	EveGet-App2 EveCon-App2 EveEnd-Process

What I have already tried is to use the KEY column as document_id but the final result is based only on the last event, that is:

Date |Key |Msj1 |Msj2 |Msj3 |Msj4 |Msj5 |Msj6 |Msj7 |Msj8						
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-04-09T13:58:22 202004091554541340000000 EveEnd-Process

I hope you can help me with ideas on how to solve this need. Thank you.

```

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [May 9, 2020, 12:39am UTC](https://discuss.elastic.co/t/use-logstash-to-transform-a-message/231811/2 "2020-05-09T00:39:59Z")

</div>

There are 3 possible approaches:

- use the [aggregate filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) to group up the documents the events having the same key in Logstash. The drawback is you can use only one worker.
- use a [scripted upsert](https://alexmarquardt.com/2019/12/17/logstash-and-elasticsearch-painless-scripted-upserts-transform-data/) to create documents having as `_id` the KEY and fields get added over time. To cope with potential documents coming out of order, it will be necessary to keep also the timestamp stored somewhere within the document
- ingest every line as a single document and use a [Transform Job](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/transforms.html) to continuously create a per-key document with a scripted metric and a group by key

---

<div class="post-metadata">

**Author:** ![dooger21](https://avatars.discourse-cdn.com/v4/letter/d/b4bc9f/32.png) [@dooger21](https://discuss.elastic.co/u/dooger21)\
**Post date:** [May 11, 2020, 3:50pm UTC](https://discuss.elastic.co/t/use-logstash-to-transform-a-message/231811/3 "2020-05-11T15:50:10Z")

</div>

Thanks Luca,

I chose the scripter upsert option and it works great.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2020, 3:50pm UTC](https://discuss.elastic.co/t/use-logstash-to-transform-a-message/231811/4 "2020-06-08T15:50:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.