5.x: Percolating with Logstash - How?

Elastic Stack Elasticsearch
1

I'm trying to build an alerting mechanism if a client is trying to access a malicious domain. I have indexed about a million "blacklist domains" to elasticsearch and trying to percolate the proxy logs against it via logstash.

Unfortunately the elasticsearch filter for logstash is not doing what i want. The elasticsearch filter should add the amount of hits (1 or 0) to the percolated event. But due some reason it's not working. I'm pretty sure that I'm understanding something wrong here, so that's probably just a misconfiguration on my side. I'm using the following setup:

Creating Index for the domains on the blacklist:

PUT /blacklist
{
    "mappings": {
        "doctype": {
            "properties": {
                "message": {
                    "type": "text"
                }
            }
        },
        "queries": {
            "properties": {
                "query": {
                    "type": "percolator"
                }
            }
        }
    }
}

Example for a single "Blacklist domain":

PUT /blacklist/queries/20000 
{ 
  "query" : { 
    "match_phrase" : { 
      "message" : "domain.xxx"  
    } 
  } 
}

Example output when searching via sense:

{
  "took": 333,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "blacklist",
        "_type": "queries",
        "_id": "2000000",
        "_score": 0.2876821,
        "_source": {
          "id": "test",
          "query": {
            "match_phrase": {
              "message": "domain.xxx"
            }
          }
        }
      }
    ]
  }
}

Logstash Config:

  elasticsearch {
    hosts       => ["localhost:9200"]
    index       => "blacklist"
    enable_sort => "false"
    query       => '"{ "percolate" : { "field" : "query", "document_type" : "doctype", "document" : { "message" : "domain.xxx" } } }"'
    fields      => { "total" => "hits" }
  }

Logstash should add the amount of hits (total) to the event adding the field "hits" with the value of total. I tried it with every field in the response, but logstash is adding nothing. What am I doing wrong here? :slight_smile:

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.