How to index a new document using existing documents?

Elastic Stack Elasticsearch
1

Hello,

I have a need to index a document, with a field that computes by adding a number to an old document's field value.

For example, I had a document (id: zoneA_userA).

{
  "_index": "online",
  "_type": "duration",
  "_id": "zoneA_userA",
...
  "_source": {
    "zone": "zoneA",
    "aggTimeSec": "1355",
...
  }
}

And I continuously receive numbers of logs saying additional online time of userA in zoneA. (Format: <user> <zone> <sec>)
userA zoneA 29

I want to add value <sec> to the document's aggTimeSec value and update the document.

Now I use logstash's elasticsearch filter to query the document, and output to same document id. This works but I wonder if this is best practice? Can I do it without querying ES from logstash? I persume the flow of logstash querying elasticsearch before output is too much time load and largely degrade logstash's performance.

Regards,
Michael

2

You should be able to do this by configuring the Elasticsearch output plugin to perform a scripted update/upsert, but I have not found any good examples showing how it is done.

3

Thanks @Christian_Dahlqvist
I refactored my logstash conf using following ES output.

     elasticsearch {
         hosts => ["..."]
         index => "online"
         document_type => "duration"
         document_id => "%{zone}_%{user}"
         manage_template => false
         action => "update"
         doc_as_upsert => true
         script_lang => "painless"
         script_type => "inline"
         script => '
             if (ctx._source.updateTimestamp != params.event.get("updateTimestamp")) {
                 ctx._source.aggTimeSec = Integer.parseInt(ctx._source.aggTimeSec) + Integer.parseInt(params.event.get("aggTimeSec"));
             }
         '
     }

Thus I can remove elasticsearch query in the filter division.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.