How to index percolator field via logstash and json_encode filter?

Elastic Stack Logstash
2

After much trial & error, I finally got it working with the following config.

input {
  ...
  jdbc {
    statement => "SELECT
        search_name
        FROM some_table
        "
    }
  }
filter {
  json_encode {
    source => "search_name"
    target => "escaped_search_name"
  }
  mutate {
    add_field => {
      "[query]" => '{
        "match" => {
          "search_name" => {
            "query" => %{escaped_search_name}
            "operator" => "and"
          }
        }
      }'
    }
  }
  json {
    source => "query"
    target => "query"
  }
  mutate {
    remove_field => [ "escaped_search_name" ]
  }
}
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "<host_name>"
  "index" => "test_percolate"
  "document_type" => "_doc"
  }
}

Turns out, I had it backwards with the json filter and json_encode filter the whole time. To pass on a structured json, the json filter should be used.

The filter parts explained:

  1. json_encode constructs escaped / encoded input before it used in the construction of json string
  2. mutate add_field: create a new field [query] with its value being a json string ( '{...}' ). It is important to have the field as [field_name], otherwise the next step to parse the string into json object doesn't work
  3. json: parse the string representation into a json object
  4. mutate remove_field: get rid of the escaped field, if it shouldn't be indexed
1 Like