Update single field in document in elastic from within logstash

Elastic Stack Logstash
1

Hi,

How can I update a single field on an existing document in elastic from logstash?

This is the docment in elastic where I want to upate field: "FEEDBACK_RECIEVED" from "NO" to "YES"

    {
  "_index": "portal-2017.07.21",
  "_type": "log",
  "_id": "100000041",
  "_version": 1,
  "_score": null,
  "_source": {
    "offset": 26230,
    "FEEDBACK_RECIEVED": "NO",
    "APPLICATION_ID": "VNW2016000041",
    "input_type": "log",
    "source": "xxx.2017-06-30.log",
    "message": "2017-05-30 10:56:30,118 INFO  [http-bio-8080-exec-9][ServiceImpl:125] Checkout registered: Registering E-Form applicationId=[[VNW2016000041]], orderId=[[100000041]].",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied",
      "UPDATE_DOCUMENT_ID",
      "SEND_LINE"
    ],
    "@timestamp": "2017-07-21T08:41:50.614Z",
    "ORDER_ID": "100000041",
    "@version": "1",
    "beat": {
      "hostname": "V2W1-HOST",
      "name": "V2W1-HOST1",
      "version": "5.4.0"
    },
    "host": "V2W1-HOST1",
    "_TIMESTAMP": "2017-05-30 10:56:30,118",
    "fields": {
      "app_id": ".portal"
    }
  },
  "fields": {
    "@timestamp": [
      1500626510614
    ]
  },
  "sort": [
    1500626510614
  ]
}

After udating in logstash output I see that the message and the tags are updated which I want to be left as is.
But "FEEDBACK_RECIEVED" which I expected to have the value "YES" is not updated.
Below is de the result after upate:

{
  "_index": "portal-2017.07.21",
  "_type": "log",
  "_id": "100000041",
  "_version": 2,
  "_score": null,
  "_source": {
    "offset": 26459,
    "FEEDBACK_RECIEVED": "NO",
    "APPLICATION_ID": "VNW2016000041",
    "input_type": "log",
    "source": "xxx.2017-06-30.log",
    "message": "2017-05-30 10:56:30,118 INFO  [http-bio-8080-exec-9][ServiceImpl:125] Checkout payment feedback: Processing payment feedback orderId=[[100000041]], transactionStatus",
    "type": "log",
    "tags": [
      "beats_input_codec_plain_applied",
      "UPDATE_FEEDBACK_RECIEVED",
      "SEND_LINE"
    ],
    "@timestamp": "2017-07-21T08:43:35.614Z",
    "ORDER_ID": "100000041",
    "@version": "1",
    "beat": {
      "hostname": "V2W1-HOST1",
      "name": "V2W1-HOST1",
      "version": "5.4.0"
    },
    "host": "V2W1-HOST1",
    "_TIMESTAMP": "2017-05-30 10:56:30,118",
    "fields": {
      "app_id": ".portal"
    },
    "YES": "NO"
  },
  "fields": {
    "@timestamp": [
      1500626615614
    ]
  },
  "sort": [
    1500626615614
  ]
}

This is the filter section where I find the old event:

elasticsearch {
	hosts => ["localhost:9200"]
	index => "portal-*"
	query => "ORDER_ID:%{ORDER_ID}"
	fields => {"FEEDBACK_RECIEVED" => "YES"}
}

And this is the output section:

elasticsearch {
	action => "update"
	document_id => "%{ORDER_ID}"
	index => "portal-%{+YYYY.MM.dd}"
	hosts => [ "localhost:9200" ]
}

Any help would greatly be appreciated.

Regards benny

2

If you need to update specific field with certain condition , so why dont you try update_by_query
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html

3

Thanks for the reply.
I need to update that field on the old event when a when the feedback event arrives in logstash.
I dont see how I can do what you suggested from within logstash.
I kind of new to the elk stack.

Do you maybe have a code example?

Regards benny

4

Okay I was missing the point in the documentation. I figured it out.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.