How to correlate events in ES

Elastic Stack Elasticsearch
1

I want to correlate events in IIS and IDS logs so that I can get the client IPs that exist in both log types in which each matches certain conditions (hits admin.aspx with 200 response code in IIS and generates critical log level in IDS).

Sample logs

{
  "src_ip": "10.0.0.1",
  "http_status": 200,
  "path": "/admin.aspx",
  "type": "iis"
},
{
  "src_ip": "10.0.0.2",
  "http_status": 200,
  "path": "/admin.aspx",
  "type": "iis"
},
{
  "src_ip": "10.0.0.1",
  "log_level": "critical",
  "type": "ids"
},
{
  "src_ip": "10.0.0.2",
  "log_level": "info",
  "type": "ids"
}

Can we write a query/agg in ES that returns only 10.0.0.1 which matches the condition I specify?

2

Anyone knows how to write this query?

3

This is what I've used in python before to merge content in two indices. Note it streams ALL docs.

mergeQuery={
   "query": {
      "match_all": { }
   },
   "sort": [
      {
         "src_ip": {
            "order": "asc"
         }
      }
   ]
}

for doc in helpers.scan(es,
                    index="indexA,indexB".
                    query=mergeQuery,
                    size=args.readsPerBulk,
                    scroll=args.maxTimeToProcessScrollPage,
                    preserve_order=True):
                    
    # Do whatever logic here to spot consecutive docs
    # with same key and merge
1 Like
4

Basically, I can accomplish this by retrieving data from ES and process. I just wonder if there're any agg or query types for similar use cases.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.