Problems doing an _update_by_query with the logstash http output plugin

Using kibana's Dev tools I have a working REST api call

     POST /unicast-prefixes-x.x.x.x/_update_by_query   
     {
       "script" : {
         "source" : "if (ctx._source.containsKey('tags')) { ctx._source.tags.add('test3') } else { ctx._source.tags = ['test3'] }"
       },
       "query" : { 
         "bool" : {
           "must" : [
             { "match" : { "TYPE" : "unicast-prefix" }}
           ],
           "filter" : [
             { "range" : { 
               "@timestamp" : { 
                   "lt" : "now"
                 }
               }
             }
           ]
         } 
       }
     }

With the following result...

{
"took": 958,
"timed_out": false,
"total": 2396,
"updated": 2396,
"deleted": 0,
"batches": 3,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": []
}

But I'm trying to implement the same API call through a logstash http output plugin and I'm having the following error in logstash.

[2018-07-08T14:09:09,933][ERROR][logstash.outputs.http ] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://elasticsearch:9200/unicast-prefixes-x.x.x.x/_search", :event=>#LogStash::Event:0x37c85594}

The logstash config is

      http {
        format => "json"
        http_method => "post"
        url => "http://elasticsearch:9200/unicast-prefixes-%{[remote_ip]}/_update_by_query"
        message => '{
          "script" : {
            "source" : "if (ctx._source.containsKey('tags')) { ctx._source.tags.add('test3') } else { ctx._source.tags = ['test3'] }"
          },
          "query" : { 
            "bool" : {
              "must" : [
                { "match" : { "TYPE" : "unicast-prefix" }}
              ],
              "filter" : [
                { "range" : { 
                  "@timestamp" : { 
                      "lt" : "now"
                    }
                  }
                }
              ]
            } 
          }
        }'
      }

Not sure what I'm doing wrong ... escaping quotes, is this a support API call for the plugin??

PS: I did the test with and without "config.support_escapes: true" in the logstash config, but same result.

Could someone help me please.
thanks

I enabled traces on elasticsearch and I got this error when the api call is executed.

org.elasticsearch.common.ParsingException: Unknown key for a VALUE_STRING in [error_text].

My problem seems quite similar to the one reported here....

Regards

I found a workaround using exec plugin and 'curl',

      exec {
        command => "
          curl -X POST \"http://elasticsearch:9200/unicast-prefixes-%{[remote_ip]}/_update_by_query?conflicts=proceed\" -H 'Content-Type:           application/json' -d'
          {
            \"script\" : {
              \"source\" : \"if (ctx._source.containsKey(params.t1)) { if (!ctx._source.tags.contains(params.t2)) { ctx._source.tags.add(params.t2) } } else { ctx._source.tags = [params.t2] }\",
              \"params\" : {
                \"t1\" : \"tags\",
                \"t2\" : \"withdrawn\"
              }
            },
            \"query\" : { 
              \"bool\" : {
                \"must\" : [
                  { \"match\" : { \"TYPE\" : \"unicast-prefix\" }}
                ],
                \"filter\" : [
                  { \"range\" : { 
                    \"@timestamp\" : { 
                        \"lt\" : \"now\"
                      }
                    }
                  }
                ]
              } 
            }
          }'
         "
      }
    }

Doing troubleshooting with curl, it was almost imposible to scape the 'tags' on (ctx._source.containsKey('tags')), and I use as workaround the params.

So, now that we have a working environment, we need to try again with http output plugin to make the most clean solution.

Regards

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.