How to progressively update my document

Elastic Stack Logstash
1

I have LS 2.0.x ES2.0 Shield 2.0 and Kibana 4.2

My configuration reads incoming application log data. The data represents all of the work flow events that our orders go thorugh - a series of pre-defined steps (approx 7 steps) that arrive at different times.

I want my Index, to contain only one document for each order. I have created in my mappings individual fields (buckets) in that index that keep track of the different work flow steps. Thus, everytime a new event comes in for that order, I want to update a different "bucket" in the document to reflect the completion of that step. Thus WFStep1 timestamp, WFStep2 timestamp, WFStep3...

I control the document_id to facilitate the updates so I get one document per order.

Thus I want to populate the document gradually as the different WorkFlow steps are completed for each order. At the end of all WF steps I want all of the 'buckets'to be populated as a result.

It almost works... the problem is that the final document only contains the very last WorkFlow step update and doesn't 'remember' any other steps that happened before it. Even though there were 7 transactions for that same order only the last one is retained in the document. Why aren't the other fields populated by prior updates preserved in the document? How can I gradually populate that document with data?

2

Hello
The default update action in logstash (and in elasticsearch) replaces the document

You want to use the upsert option:


https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html

If you want to only have completed orders in your elasticsearch, you can also use the aggregate filter in logstash. This should do the same trick, without writing to the database 7 times per order
https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

I hope this helps

3

This is what I have setup, but still no dice:

    elasticsearch {
            hosts => localhost
            index => "orders_alt"
            action => "update"
            user =>     "es_admin"
            password => "pa$$word"
            document_id => "%{env}_%{orderNumber}"
            doc_as_upsert => true
    }

That's what the examples showed - why doesn't that preserve the data following incremental updates to the same document?

4

Hi
I am facing a similar situation , need to update the documents based on the document ID, but its replacing the existing document with the new document with same doc_id.
Please help! Thanks in Advance.

5

Check your "action", it should be "upsert", not "update".