Problems doing an _update_by_query with the logstash http output plugin

[Efrain_Gonzalez]

Using kibana's Dev tools I have a working REST api call

     POST /unicast-prefixes-x.x.x.x/_update_by_query   
     {
       "script" : {
         "source" : "if (ctx._source.containsKey('tags')) { ctx._source.tags.add('test3') } else { ctx._source.tags = ['test3'] }"
       },
       "query" : { 
         "bool" : {
           "must" : [
             { "match" : { "TYPE" : "unicast-prefix" }}
           ],
           "filter" : [
             { "range" : { 
               "@timestamp" : { 
                   "lt" : "now"
                 }
               }
             }
           ]
         } 
       }
     }

With the following result...

{
"took": 958,
"timed_out": false,
"total": 2396,
"updated": 2396,
"deleted": 0,
"batches": 3,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": []
}

But I'm trying to implement the same API call through a logstash http output plugin and I'm having the following error in logstash.

[2018-07-08T14:09:09,933][ERROR][logstash.outputs.http ] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://elasticsearch:9200/unicast-prefixes-x.x.x.x/_search", :event=>#LogStash::Event:0x37c85594}

The logstash config is

      http {
        format => "json"
        http_method => "post"
        url => "http://elasticsearch:9200/unicast-prefixes-%{[remote_ip]}/_update_by_query"
        message => '{
          "script" : {
            "source" : "if (ctx._source.containsKey('tags')) { ctx._source.tags.add('test3') } else { ctx._source.tags = ['test3'] }"
          },
          "query" : { 
            "bool" : {
              "must" : [
                { "match" : { "TYPE" : "unicast-prefix" }}
              ],
              "filter" : [
                { "range" : { 
                  "@timestamp" : { 
                      "lt" : "now"
                    }
                  }
                }
              ]
            } 
          }
        }'
      }

Not sure what I'm doing wrong ... escaping quotes, is this a support API call for the plugin??

PS: I did the test with and without "config.support_escapes: true" in the logstash config, but same result.

Could someone help me please.
thanks

[Efrain_Gonzalez]

I enabled traces on elasticsearch and I got this error when the api call is executed.

org.elasticsearch.common.ParsingException: Unknown key for a VALUE_STRING in [error_text].

[Efrain_Gonzalez]

My problem seems quite similar to the one reported here....

How to update a document in elasticsearch with logstash http output plugin

Hi all.... Here i am trying to update a document in elasticsearch with the help of output logstash http plugin here is my logstash conf input { elasticsearch { hosts => "192.168.1.59:9233" index => "titanic_kfk_shrink" query => '{ "query":{ "match" : { "PassengerId" : "200" } }, "sort": [ "_doc" ] }' size => 5000 scroll => "5m" docinfo => true docinfo_target => "@metadata" } } output { stdout { codec => rubydebug { metadata => true } }…

Regards

[Efrain_Gonzalez]

I found a workaround using exec plugin and 'curl',

      exec {
        command => "
          curl -X POST \"http://elasticsearch:9200/unicast-prefixes-%{[remote_ip]}/_update_by_query?conflicts=proceed\" -H 'Content-Type:           application/json' -d'
          {
            \"script\" : {
              \"source\" : \"if (ctx._source.containsKey(params.t1)) { if (!ctx._source.tags.contains(params.t2)) { ctx._source.tags.add(params.t2) } } else { ctx._source.tags = [params.t2] }\",
              \"params\" : {
                \"t1\" : \"tags\",
                \"t2\" : \"withdrawn\"
              }
            },
            \"query\" : { 
              \"bool\" : {
                \"must\" : [
                  { \"match\" : { \"TYPE\" : \"unicast-prefix\" }}
                ],
                \"filter\" : [
                  { \"range\" : { 
                    \"@timestamp\" : { 
                        \"lt\" : \"now\"
                      }
                    }
                  }
                ]
              } 
            }
          }'
         "
      }
    }

Doing troubleshooting with curl, it was almost imposible to scape the 'tags' on (ctx._source.containsKey('tags')), and I use as workaround the params.

So, now that we have a working environment, we need to try again with http output plugin to make the most clean solution.

Regards