Using search output to drive a new search

[DunkB]

Hi All,
Not sure if this belongs in Kibana or Elastic but here goes.

I've got some log data (openldap) in elasticsearch but as single lines, so the connect IP is on a different line from the bind command is on a different line from the TLS negotiation...

I can user connectionID to link these records, so a search for say connections from 192.168.0.1 and only return connectionID. If I then search by the returned connectionID instead of IP then I get the transactions for the IP.

Time 	connection    	operation_type      	openldap_message    
September 13th 2018, 12:47:50.409	405248	ACCEPT 	from IP=192.168.0.1:1234 (IP=0.0.0.0:636)
September 13th 2018, 12:47:50.417	405248	TLS 	established tls_ssf=128 ssf=128
September 13th 2018, 12:47:50.418	405248	BIND 	dn="" method=128
September 13th 2018, 12:47:50.418	405248	RESULT 	tag=97 err=0 text=
September 13th 2018, 12:47:50.418	405248	SRCH 	base="dc=example,dc=com" scope=2 deref=0 filter="(uid=monitor)"
September 13th 2018, 12:47:50.477	405248	SEARCH RESULT 	tag=101 err=0 nentries=0 text=
September 13th 2018, 12:47:50.478	405248	UNBIND	 - 
September 13th 2018, 12:47:50.478	405248	 - 	closed

Is there any way to make this easier / automate it a bit so that I can search for all the connectionIDs associated with an IP address and return the records for those connectionIDs?

Buckets or scripting look like possibilities but I'm not sure where to start (point me at logs / demos/ training please

Thanks,
Duncan

[Aaron_Caldwell]

Hello Duncan,

This could likely be solved with a terms aggregation:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

If I'm understanding correctly, the relevant term would be connection, and you could call the aggregation anything you like, connections would work.

As for training/docs, I'd start with the docs I've linked above if you'd like to learn more about Elasticsearch. If you'd like to focus on Kibana, the Kibana docs walk you through thing sequentially and even provide sample data:

https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html

Beyond this, I find our blogs to be an excellent learning source, here are some focused on aggregations:

https://www.elastic.co/search?q=aggregations&section=Learn%2FBlog

Hope this helps!

Regards,
Aaron

[DunkB]

Thanks Aaron,
That looked promising but it seems you can't return the contents of a bucket? So having split the index by connection, and created a bucket for each connection, I'd want to search for field=foo within the buckets and return the message field for every document within each bucket where a document in the bucket (there would be only 1 document containing the field foo per bucket) contained the string I'm looking for which may be an IP or TLS error or modify on a specific field.

I'm starting to think a better question may be how to store the openldap logs. I'll ask something in the logstash forum.

Cheers,
Duncan