Issue with logstash input elasticsearch plugin with ES 5.4

Elastic Stack Elasticsearch
1

HI this conf works with ES 5.3

input {
  elasticsearch {
    hosts => "http://localhost:9200"
	index => "sirene"
	query => '{"query": {"query_string" : {"query": "(CODGEO:60471 AND provider:sp_mairie)"}}}'
  }
}
output {
  stdout { codec => json }
}

But fail with ES 5.4 with this error:

Error: [400] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Failed to parse request body"}],"type":"illegal_argument_exception","reason":"Failed to parse request body","caused_by":{"type":"json_parse_exception","reason":"Unrecognized token 'DnF1ZXJ5VGhlbkZldGNoBQAAAAAAABHMFm84Mm84SFpzU3VHSzdWMHdWQ3N3NGcAAAAAAAARzhZvODJvOEhac1N1R0s3VjB3VkNzdzRnAAAAAAAAEc8WbzgybzhIWnNTdUdLN1Ywd1ZDc3c0ZwAAAAAAABHQFm84Mm84SFpzU3VHSzdWMHdWQ3N3NGcAAAAAAAARzRZvODJvOEhac1N1R0s3VjB3VkNzdzRn': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@d38650b; line: 1, column: 457]"}},"status":400}

Thanks for your help

2

I would change your query to be regular syntax and not use query_string syntax. It's not listed as a breaking change for 5.4, but I think that they're getting away from using Lucene query_string as it has a profound ability to do bad things to clusters.

If you were to run the same query against 5.4 from the command-line (via curl, for instance) you would get the same result (a 400 error), I think. As such, this may be more a question for the Elasticsearch forum than the Logstash one.

3

Hi!

With curl it's working (so I guess it's a logstash question)...
I will try to change the query in order not to use the query_string (this is still in the 5.4 docs https://www.elastic.co/guide/en/elasticsearch/reference/5.4/query-dsl-query-string-query.html) . What format do you recommend?

Best

4

Hmm. But it is Elasticsearch that is responding with the 400 error...

Regardless, I would use filters, first and foremost, as you are looking for exact term values. Performing a query instead of a filter here is less efficient. An example is here.

5

Thanks I will try this way! And move this post to the elastisearch forum

6

resolved here https://github.com/logstash-plugins/logstash-input-elasticsearch/issues/66

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.