Elasticsearch _update_by_query in logstash error [HTTP Output Failure] Encountered non-2xx HTTP code 400

Elastic Stack Logstash
1

Properly working _update_by_query call -

POST /s1test-demo7/_update_by_query
{
  "script": {
      "source": "ctx._source.externaldata = params.externaldata",
      "lang": "painless",
      "params": {
        "externaldata":{
          "field1": "1",
          "field2": "abc"  
        }
      }
    },
  "query": {
    "bool": {
      "must": [
        { "match": { "h.req-id": "Test9GrpA"} },
        { "match": { "h.process-code": "DemoS99"} }
      ]
    }
  }
}

This API actually adds "externaldata" field with 2 inner fields as a nested json correctly in query matching existing docs.
I need help with logstash pipeline to do the same -

input {
    file {
        id => "updatedata"
        path => "D:/p3-test1.json"
        start_position => "beginning"
        sincedb_path => "D:/sdb/p3-test1.sdb"
    }
}
filter {
    json {
        source => "message"
    }
    mutate {
        add_field => {
            "[script][lang]" => "painless"
            "[script][source]" => "ctx._source.externaldata = params.externaldata"
            "[script][params][externaldata][field1]" => "%{field1}"
            "[script][params][externaldata][field2]" => "%{field2}"
            "[query][bool][must][match][h.req-id]" => "%{req-id}"
            "[query][bool][must][match][h.process-code]" => "%{process-code}"
        }
    }
}
output {
    stdout {
        codec => rubydebug
    }
    http {
        url => "http://localhost:9200/s1test-demo7/_update_by_query"
        headers => { "Authorization" => "Basic ZWxhc3RpYzplbGFzdGlj" }
        http_method => "post"
        format => "json"
    }
}

The logstash pipeline must include authentication, otherwise I get HTTP error 401. But I am not sure if syntax is correct here. The documentation and elasticsearch forum posts has variations in http output plugin headers.
Logstash output -

[2021-05-29T20:14:13,226][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
            "host" => "mypc",
        "@version" => "1",
          "req-id" => "Test9GrpA",
          "field2" => "default",
      "@timestamp" => 2021-05-29T14:44:13.521Z,
          "field1" => "1",
           "query" => {
        "bool" => {
            "must" => {
                "match" => {
                    "h.process-code" => "DemoS99",
                          "h.req-id" => "Test9GrpA"
                }
            }
        }
    },
         "message" => "{\"req-id\":\"Test9GrpA\",\"process-code\":\"DemoS99\",\"field1\":\"1\",\"field2\":\"default\"}\r",
    "process-code" => "DemoS99",
          "script" => {
          "lang" => "painless",
        "params" => {
            "externaldata" => {
                "field2" => "default",
                "field1" => "1"
            }
        },
        "source" => "ctx._source.externaldata = params.externaldata"
    },
            "path" => "D:/p3-test1.json"
}
[2021-05-29T20:14:14,268][ERROR][logstash.outputs.http    ][main][2bc46338fec26d73c819043dac159f1d12397fefc800c84c7d2e0f0d16b278c2] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/s1test-demo7/_update_by_query", :event=>#<LogStash::Event:0x2badd231>}

Please help.

2

It has been long but no reply received. May be scripting isnt popular.
I solved this issue by shifting filter script section to output http section.
My answer - https://stackoverflow.com/a/67848259/10099219

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.