Error: Encountered non-2xx HTTP code 400

Elastic Stack Logstash
1

Hi,
I'm using the HTTP output plugin to delete some documents by query.
Here's my config file:

input {
  jdbc {
    jdbc_driver_library => "/root/infr/logstash-8.12.2/connector/mysql-connector-j-8.3.0.jar"
    jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://l"
    jdbc_user => "xx"
    jdbc_password => "xx"
    jdbc_paging_enabled => true
    tracking_column => "unix_ts_in_secs"
    use_column_value => true
    tracking_column_type => "numeric"
    schedule => "*/10 * * * * *"
    statement => "SELECT build_id FROM xxx"
  }
}
filter {
  mutate {
    remove_field => ["unix_ts_in_secs"]
  }
}
output {
  http {
    url => "xx:9200/index_name/_delete_by_query"
    http_method => "post"
    format => "json"
    user => "xx"
    password => "xx"
    headers => {
      "Content-Type" => "application/json"
    }
    message => '{"query": {"match": {"build_id": "%{[build_id]}"}}}'
    retry_non_idempotent => false
  }
}

Got this error log:

Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"xxx", :event=>#<LogStash::Event:0x30854c26>}

400 means a bad request, so is this caused by the wrong config for dynamic field ${[build_id]} in the "message" part?

2

My reading of the code is that the message option is ignored if the format option is json. I think you should be using format => "message". The documentation seems to say that too.

3

Hi,
After changing to format => "message", got 409 return code:

Encountered non-2xx HTTP code 409
4

delete_by_query does a search for documents that match the query, then it iterates over that list of documents and deletes them. But it does a version check to make sure that the document has not been updated between the query and the delete. If it has been updated then it returns a 409 error and does not delete the document.

If you want to test the output then you can find a build_id that you want to delete from elasticsearch, and replace your jdbc input with

input { generator { count => 1 message => '{"build_id": "XXX"}' codec => json } }
5

Hi,
I'm sure there will only be a deleting request sent during this process, as it is just my local testing environment.

6

Then I am out of ideas. I do not run elasticsearch, just logstash. Perhaps @stephenb or one of the other regulars can help.

7

I think message is correct.

I think debugging the 409 is the next step.

I would do what @Badger suggested to debug and isolate.

Try 1 single known id.

8

Hi,
I just tried and got a 406 return code.