Regroupe LDAP log lines with same id on the same DataTable

(Noureddine Brahmi) #1


I'm currently working on an ElasticSearch project that monitors LDAP logs.

In the LDAP logs, related events shares the same id (connection_number). I was wondering is there is a solution to regroupe the various fieds in different documents that shares the same id.


[04/Aug/2018:22:34:15 +0200] conn=184214 op=0 msgId=1 - BIND dn="cn=XOPOY,ou=programs,o=psa" method=128 version=3

[04/Aug/2018:22:34:15 +0200] conn=184214 op=0 msgId=1 - RESULT err=0 tag=97 nentries=0 etime=0.000450 dn="cn=mwplim02,ou=programs,o=psa"

[04/Aug/2018:22:34:15 +0200] conn=184214 op=2 msgId=3 - UNBIND

The goal is to have a datatable with:

TIMESTAMP | conn | dn (from the BIND) | err | etime

PS: I already have a grok filter in place that sperates the various fields

Thank you !

(I already posted this question in the Kibana Forum, and I was asked to rather ask it here)

(Lewis Barclay) #2

Would the aggregate filter work?

(Noureddine Brahmi) #4

It might but the problem is :

Most of the time between a BIND and an UNBIND thousands of lines and operation would come in between, and even other conn with different conn IDs. So I don't know if it's a viable solution.

I thought about adding the dn ( from the BIND ) to each line that has the same conn ID. But I don't if/how I can implement it in logstash.

(Lewis Barclay) #5

Is there a clear start and end event though? If so then aggregate should work?

(Noureddine Brahmi) #7

Yes there is a clear start and end .

The start event is operation_type = BIND, the end is operation_type = UNBIND.

(the seperated fields of each log line are already in place)

(Lewis Barclay) #8

So do you not think aggregate would work?