Using upsert to update existing record base on value in fields

Hi all,

Currently I'm trying to manage the Elasticsearch records from Filebeat which came from bash script and write to CSV file for each execute, then Filebeat collect log from CSV file.
Below are example of header(not including in file)

And the log file are as below. (came from bash script each execute/line), 1, check file A, False, 2, check file B, True, 1, check file A, True

I used Filebeat processor dissect to separate the field and directly ship to Elasicsearch without Logstash for now. The problem I'm facing now is record 1 and 3 are the same server_ip and same task except the result that I would like to update instead and if server_ip, task_number, task_name are new, so insert them.
I know that Filebeat itself can't do this because it just shipping the log base on the message writing into CSV file

Currently in Elasticsearch
record1 ----, 1, check file A, False
record2 ----, 2, check file B, True
record3 ----, 1, check file A, True

The expected
record1 ----, 1, check file A, True
record2 ----, 2, check file B, True

The SQL query is like below
UPDATE dbName SET result = "True" WHERE server_ip = "" AND task_number = "1" AND task_name = "check file A"

INSERT INTO dbNAME (server_ip, task_number, task_name, result) VALUES ("", "1", "check file B", "True")

Is Logstash be able to use upsert to update record base on value of multiple fields like this? Or any idea I can handle on this one?

I've a query for _update_by_query below, This help me achived what I want.

curl -X POST -k -u myusername:mypassword "https://localhost:9200/my-index/_update_by_query" -H 'Content-Type: application/json' -d'
  "script": {
    "inline": "ctx._source.result = 'True'",
    "lang": "painless"
  "query": {
      "bool": {
            "must": [
                     {"match": {"server_ip": ""}},
                     {"match": {"task_number": "1"}},
                     {"match": {"task_name": "check file A"}}

but then I would like to convert them into logstash output so I tried but still doesn't work

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    ssl_certificate_verification => 'false'
    index => 'my-index'
    user => 'myuser'
    password => 'mypassword'
    doc_as_upsert => true
    script => 'if(ctx._source.server_ip == "" and ctx._source.task_number == "1" ctx._source.task_name == "check file A") { should update the result to "True" here}; else (ctx._source.item.add)'
    action => "create"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.