Regroupe LDAP log lines with same id on the same DataTable

[Noureddine_Brahmi]

Hello,

I'm currently working on an ElasticSearch project that monitors LDAP logs.

In the LDAP logs, related events shares the same id (connection_number). I was wondering is there is a solution to regroupe the various fieds in different documents that shares the same id.

Ex:

[04/Aug/2018:22:34:15 +0200] conn=184214 op=0 msgId=1 - BIND dn="cn=XOPOY,ou=programs,o=psa" method=128 version=3

[04/Aug/2018:22:34:15 +0200] conn=184214 op=0 msgId=1 - RESULT err=0 tag=97 nentries=0 etime=0.000450 dn="cn=mwplim02,ou=programs,o=psa"

[04/Aug/2018:22:34:15 +0200] conn=184214 op=2 msgId=3 - UNBIND

The goal is to have a datatable with:

TIMESTAMP | conn | dn (from the BIND) | err | etime

PS: I already have a grok filter in place that sperates the various fields

Thank you !

(I already posted this question in the Kibana Forum, and I was asked to rather ask it here)

[Eniqmatic (Lewis Barclay)]

Would the aggregate filter work?

https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

[Noureddine_Brahmi]

It might but the problem is :

Most of the time between a BIND and an UNBIND thousands of lines and operation would come in between, and even other conn with different conn IDs. So I don't know if it's a viable solution.

I thought about adding the dn ( from the BIND ) to each line that has the same conn ID. But I don't if/how I can implement it in logstash.

[Eniqmatic (Lewis Barclay)]

Is there a clear start and end event though? If so then aggregate should work?

[Noureddine_Brahmi]

Yes there is a clear start and end .

The start event is operation_type = BIND, the end is operation_type = UNBIND.

(the seperated fields of each log line are already in place)

[Eniqmatic (Lewis Barclay)]

So do you not think aggregate would work?