Logstash Elasticsearch Filter Lookup No Results

[MakoWish (Eric Christensen)]

I am having an issue with the Logstash Elasticsearch filter plugin not returning results, when I know there should be. I am sure it is an escape character, or something along those lines, but I cannot for the life of me figure it out.

To duplicate the issue:

PUT index_name/_doc/1
{
  "user": {
    "display_name": "Smith, John",
    "distinguished_name": "DN=Smith\\, John,OU=Users,DC=contoso,DC=com"
  }
}

I can confirm the manager's record now exists in the index:

GET index_name/_doc/1
{
  "_index" : "index_name",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 1930378,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "user" : {
      "display_name" : "Smith, John",
      "distinguished_name" : """DN=Smith\, John,OU=Users,DC=contoso,DC=com"""
    }
  }
}

I have an Elasticsearch filter plugin in the pipeline to look up a manager's display name for every user that is ingested.

filter{
  if [user][manager][distinguished_name] {
    elasticsearch {
      hosts => [
        "https://ES_Host.Contoso.com:9200"
      ]
      user => "foo"
      password => "${secret}"
      index => "index_name"
      query => 'user.distinguished_name: "%{[user][manager][distinguished_name]}"'
      fields => {"[user][display_name]" => "[user][manager][display_name]"}
      tag_on_failure => ["_manager_lookup_failure"]
    }
  }
}

Given the following record going through the Logstash pipeline, I would expect the plugin to find Joe's manager's display name:

{
  "user": {
    "display_name": "Dirt, Joe",
    "distinguished_name": "DN=Dirt\\, Joe,OU=Users,DC=contoso,DC=com",
    "manager": {
      "distinguished_name": "DN=Smith\\, John,OU=Users,DC=contoso,DC=com"
    }
  }
}

Unfortunately, there are no errors, tag_on_failure is not added to Joe's document in Elastic, and I am not getting any result.

I have tried replacing the single quotes in the query with double quotes (and escaping the double quotes around the manager's \"DN\") to no avail.

I set Logstash to debug logging, and I can see the query being passed. If I copy and paste the exact query from the Logstash logs into Kibana, the manager's document does come up. Why does the plugin not find it?

I would sincerely appreciate any assistance in figuring this one out.

[MakoWish (Eric Christensen)]

Disregard. It was the backslash in the distinguished name needing to be escaped.

Solution:

filter{
  if [user][manager][distinguished_name] {
    ruby {
      code => "
        event.set('[_temp][manager][distinguished_name]', event.get('[user][manager][distinguished_name]').gsub(/\\/, '\\\\\\'))
      "
    }

    elasticsearch {
      hosts => [
        "https://ES_Host.Contoso.com:9200"
      ]
      user     => "foo"
      password => "${secret}"
      index    => "index_name"
      query    => 'user.distinguished_name: "%{[_temp][manager][distinguished_name]}"'
      fields   => {"[user][name]"      => "[user][manager][name]"}
      fields   => {"[user][full_name]" => "[user][manager][full_name]"}
      tag_on_failure => ["_manager_lookup_failure"]
    }

    mutate {remove_field => ["[_temp]"]}
  }
}