Number_format_exception: Elasticsearch Query Template - check if value of int field present inside array[int] field

[tinhn (Tinh)]

Hello,

I'm trying to write a logstash filter that reads from an elasticsearch index using a template. I want to check if an integer matches a value stored in an array of ints.

// incoming record
{
  "id": 2
}

// query template checking if id is present in the fields `to_ids` or `from_ids`
{
  "query": {
    "bool": {
      "should": [
        {
           "match": {
               "to_ids": "%{[id]}"
            }
        },
        {
           "match": {
               "from_ids": "%{[id]}"
            }
        }
      ]
    }
  }
}

// example of document it will match with
{
  to_ids: [2, 44]
  from_ids: [1]
}

The query works when I add it as a filter in Kibana Discover (replacing %{[id]} with string or int values), but fails as a logstash filter. Logstash spits out this error message

Failed to query elasticsearch for previous event {:index=>"...", :error=>"[400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: For input string: \"%{[id]}\"","index_uuid":"FXZRQa1xTt6jQ44zbeMZvg","index":"..."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"tenant-service-revised","node":"cUnu_wtqTBieVb9Jrgi_rw","reason":{"type":"query_shard_exception","reason":"failed to create query: For input string: \"%{[id]}\"","index_uuid":"FXZRQa1xTt6jQ44zbeMZvg","index":"...","caused_by":{"type":"number_format_exception","reason":"For input string: \"%{[id]}\""}}}]},"status":400}"}

[Badger]

Are you using a filter to query elasticsearch? What does the filter configuration look like?

The error message comes from elasticsearch, which means that logstash did not interpolate the value of the [id] field (which it tries to for both templates and query strings.). That very much suggests that the [id] field does not exist.

[tinhn (Tinh)]

Hello @Badger,

Thanks for your response. Yes, I'm using the elasticsearch filter plugin. I set up my filter like so (just adding some fields when I see a match).

elasticsearch {
  id => "example-filter"
  hosts => ["http://localhost:9200"]
  index => "example-index"
  query_template => "<PATH_TO_TEMPLATE>/example.template.json"
  fields => {
    "description" => "user.service_description"
    "name" => "user.name"
  }
  enable_sort => false
}

I checked that the id field exists by outputting to stdout.

I believe the error is caused when using the match operation with a field type int instead of string when using query templates. To verify this, I added a filter to convert the id field from int --> string and referenced the string field in my template. This change works despite matching against an array of ints. Weirdly enough, adding the query as a filter via Kibana Discover also works when setting id as int AND string.