Error in logstash output by using elasticsearch update by query

Elastic Stack Logstash
1

Hi All,

I am trying to update new field to the existing documents using update by query through logstash and below is the configuration I have tried,

input {
   http_poller {
     urls => {
       version => "https://dev/auth/version"
     }
     request_timeout => 60
     schedule => { cron => "* * * * *"}
     codec => "plain"
     metadata_target => "http_poller_metadata"
     add_field => { "log_type" => "AUTH_DEV" }
   }
}
filter {
  if [log_type] == "AUTH_DEV" {
    mutate {
      add_field => {
        "version" => "%{message}"
      }
    remove_field => ["@version","http_poller_metadata","message"]
    }
  }
}
output {
  http {
    url => "https://localhost:9200/test/_update_by_query"
    headers => { "Authorization" => "Basic ZWxhc3RpYzpXTUBlbGDejia==" }
    http_method => "post"
    format => "message"
    content_type => "application/json"
    message => '{
            "script": {
              "source": "ctx._source.number= params.number",
              "lang": "painless",
              "params": {
              "number":{
                "number": "%{version}",
              }
             }
            },
            "query": {
              "term": {
                "app_module": "%{log_type}"
              }
             }
            }'
  }
  stdout {
    codec => rubydebug
  }
}

When I execute the below command we are getting HTTP Code 400 error in the logs,

/apps/logstash/bin/logstash -f /opt/logstash/fileinput/make.conf --path.settings=/opt/logstash --java-execution --path.data=/opt/logstash/SIEM/

[2022-09-14T19:20:52,897][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x4ef287de run>"}
[2022-09-14T19:20:54,859][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>1.95}
[2022-09-14T19:20:55,252][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-09-14T19:20:55,324][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>2.48}
[2022-09-14T19:20:55,362][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * *"}, :timeout=>nil}
[2022-09-14T19:20:55,427][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2022-09-14T19:20:55,568][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
/opt/logstash-7.16.3/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
[2022-09-14T19:21:01,200][ERROR][logstash.outputs.http    ] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"https://localhost:9200/test/_update_by_query", :event=>#<LogStash::Event:0x48341256>}
{
      "log_type" => "AUTH_DEV",
    "@timestamp" => 2022-09-14T11:21:00.869Z,
       "version" => "0.0.4-SNAPSHOT"
}

Though I verified the authorization token using curl in the same server and its working fine,

curl --request GET 'https://localhost:9200' --header 'Authorization: Basic ZWxhc3RpYzpXTUBlbGDejia==' --header 'Content-Type:application/json'

{
  "name" : "localhost",
  "cluster_name" : "logging-dev",
  "cluster_uuid" : "amsRwPiu-CaOYHvzama",
  "version" : {
    "number" : "7.16.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "dkfe4eab2297e949ecsdfsdd4629sdf",
    "build_date" : "2022-01-06T23:43:02.825887787Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Please share your thoughts on the issue and it would be really helpful.

Thanks,
Ganeshbabu R

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.