# Correlation between events

**URL:** <https://discuss.elastic.co/t/correlation-between-events/128789>\
**Category:** Elasticsearch\
**Created:** [April 20, 2018, 4:26am UTC](https://discuss.elastic.co/t/correlation-between-events/128789 "2018-04-20T04:26:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sylvain-69](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain-69/32/26561_2.png) [@Sylvain-69](https://discuss.elastic.co/u/Sylvain-69)\
**Post date:** [April 20, 2018, 4:26am UTC](https://discuss.elastic.co/t/correlation-between-events/128789/1 "2018-04-20T04:26:18Z")

</div>

Hello everybody !

Let say I send into elasticsearch those events :

Fri 2018/04/20 03:04:09 gammu-smsd[29236]: New message to send: 59389  
Fri 2018/04/20 03:04:09 gammu-smsd[29236]: Message without SMSC, assuming you want to use the one from phone  
Fri 2018/04/20 03:04:11 gammu-smsd[29236]: SMS sent on device: "/dev/ttyACM0" status=21, reference=-1  
Fri 2018/04/20 03:04:11 gammu-smsd[29236]: Error getting send status of message: Unknown error. (UNKNOWN[27])

where the [29236] is the "ID" of all the events who concerns this text message.

Is it possible to retrieve the phone number (here it is 59389) to display it in a dashboard where I could see the number of all the sms who was sent with an error ?

Maybe by creating a new document for each message sent and then update this document with a new field with error or OK when we know about the result ?

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [April 27, 2018, 1:34pm UTC](https://discuss.elastic.co/t/correlation-between-events/128789/2 "2018-04-27T13:34:51Z")

</div>

It'll take some pre-processing to make this work reliably. I'd take a look at [Logstash's grok filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html), of the [ingest grok processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html).

Grok is used to transform log messages into structured json documents, using a variety of patterns and regex. Once you have the data in some kind of structured document, it becomes a lot easier to do what you're asking in Elasticsearch.

You could try without pre-processing the log lines, but you'll be limited to full-text search and will likely match many unrelated documents.

---

<div class="post-metadata">

**Author:** ![Sylvain-69](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain-69/32/26561_2.png) [@Sylvain-69](https://discuss.elastic.co/u/Sylvain-69)\
**Post date:** [May 3, 2018, 12:12am UTC](https://discuss.elastic.co/t/correlation-between-events/128789/3 "2018-05-03T00:12:58Z")

</div>

Thank you for your answer.  
It's actualy the next step I'm looking for, after I processed the syslog messages and that I did the mapping with all my different fields.

How can I visualize in kibana that this particuliar sms was sent with an error to this phone number ? Since all those information come in different syslog messages.

Is it possible to create a new document that we update during the processing of the logs ?

for example, logstash received this syslog message :

Fri 2018/04/20 03:04:09 gammu-smsd[29236]: New message to send: 59389

We use grok or else and extract the sms\_ID : 29236, and the phone number : 59389 and we put those information into another document, where the sms\_ID will be used to identify the document in elasticsearch and add the phone number.

Later when we receive the status (error or sent) :

Fri 2018/04/20 03:04:11 gammu-smsd[29236]: Error getting send status of message: Unknown error. (UNKNOWN[27])

we update the document we created by extracting the same sms\_ID and then add the status error or sent with the error message if needed.

That way I can visualize in kibana which messages was sent with an error to which number.

is it possible to do that in the logstash filter configuration ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 31, 2018, 12:13am UTC](https://discuss.elastic.co/t/correlation-between-events/128789/4 "2018-05-31T00:13:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.