Correlation between events

Elastic Stack Elasticsearch
1

Hello everybody !

Let say I send into elasticsearch those events :

Fri 2018/04/20 03:04:09 gammu-smsd[29236]: New message to send: 59389
Fri 2018/04/20 03:04:09 gammu-smsd[29236]: Message without SMSC, assuming you want to use the one from phone
Fri 2018/04/20 03:04:11 gammu-smsd[29236]: SMS sent on device: "/dev/ttyACM0" status=21, reference=-1
Fri 2018/04/20 03:04:11 gammu-smsd[29236]: Error getting send status of message: Unknown error. (UNKNOWN[27])

where the [29236] is the "ID" of all the events who concerns this text message.

Is it possible to retrieve the phone number (here it is 59389) to display it in a dashboard where I could see the number of all the sms who was sent with an error ?

Maybe by creating a new document for each message sent and then update this document with a new field with error or OK when we know about the result ?

2

It'll take some pre-processing to make this work reliably. I'd take a look at Logstash's grok filter plugin, of the ingest grok processor.

Grok is used to transform log messages into structured json documents, using a variety of patterns and regex. Once you have the data in some kind of structured document, it becomes a lot easier to do what you're asking in Elasticsearch.

You could try without pre-processing the log lines, but you'll be limited to full-text search and will likely match many unrelated documents.

3

Thank you for your answer.
It's actualy the next step I'm looking for, after I processed the syslog messages and that I did the mapping with all my different fields.

How can I visualize in kibana that this particuliar sms was sent with an error to this phone number ? Since all those information come in different syslog messages.

Is it possible to create a new document that we update during the processing of the logs ?

for example, logstash received this syslog message :

Fri 2018/04/20 03:04:09 gammu-smsd[29236]: New message to send: 59389

We use grok or else and extract the sms_ID : 29236, and the phone number : 59389 and we put those information into another document, where the sms_ID will be used to identify the document in elasticsearch and add the phone number.

Later when we receive the status (error or sent) :

Fri 2018/04/20 03:04:11 gammu-smsd[29236]: Error getting send status of message: Unknown error. (UNKNOWN[27])

we update the document we created by extracting the same sms_ID and then add the status error or sent with the error message if needed.

That way I can visualize in kibana which messages was sent with an error to which number.

is it possible to do that in the logstash filter configuration ?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.