Properly working _update_by_query call -
POST /s1test-demo7/_update_by_query
{
"script": {
"source": "ctx._source.externaldata = params.externaldata",
"lang": "painless",
"params": {
"externaldata":{
"field1": "1",
"field2": "abc"
}
}
},
"query": {
"bool": {
"must": [
{ "match": { "h.req-id": "Test9GrpA"} },
{ "match": { "h.process-code": "DemoS99"} }
]
}
}
}
This API actually adds "externaldata" field with 2 inner fields as a nested json correctly in query matching existing docs.
I need help with logstash pipeline to do the same -
input {
file {
id => "updatedata"
path => "D:/p3-test1.json"
start_position => "beginning"
sincedb_path => "D:/sdb/p3-test1.sdb"
}
}
filter {
json {
source => "message"
}
mutate {
add_field => {
"[script][lang]" => "painless"
"[script][source]" => "ctx._source.externaldata = params.externaldata"
"[script][params][externaldata][field1]" => "%{field1}"
"[script][params][externaldata][field2]" => "%{field2}"
"[query][bool][must][match][h.req-id]" => "%{req-id}"
"[query][bool][must][match][h.process-code]" => "%{process-code}"
}
}
}
output {
stdout {
codec => rubydebug
}
http {
url => "http://localhost:9200/s1test-demo7/_update_by_query"
headers => { "Authorization" => "Basic ZWxhc3RpYzplbGFzdGlj" }
http_method => "post"
format => "json"
}
}
The logstash pipeline must include authentication, otherwise I get HTTP error 401. But I am not sure if syntax is correct here. The documentation and elasticsearch forum posts has variations in http output plugin headers.
Logstash output -
[2021-05-29T20:14:13,226][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
"host" => "mypc",
"@version" => "1",
"req-id" => "Test9GrpA",
"field2" => "default",
"@timestamp" => 2021-05-29T14:44:13.521Z,
"field1" => "1",
"query" => {
"bool" => {
"must" => {
"match" => {
"h.process-code" => "DemoS99",
"h.req-id" => "Test9GrpA"
}
}
}
},
"message" => "{\"req-id\":\"Test9GrpA\",\"process-code\":\"DemoS99\",\"field1\":\"1\",\"field2\":\"default\"}\r",
"process-code" => "DemoS99",
"script" => {
"lang" => "painless",
"params" => {
"externaldata" => {
"field2" => "default",
"field1" => "1"
}
},
"source" => "ctx._source.externaldata = params.externaldata"
},
"path" => "D:/p3-test1.json"
}
[2021-05-29T20:14:14,268][ERROR][logstash.outputs.http ][main][2bc46338fec26d73c819043dac159f1d12397fefc800c84c7d2e0f0d16b278c2] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/s1test-demo7/_update_by_query", :event=>#<LogStash::Event:0x2badd231>}
Please help.