Logstash output Elastic upsert

Elastic Stack Logstash
1

Hi Team,

i am looking for clarity on logstash's Elasticsearch output attribute docs_as_upsert and action => update.
Now for this action to work properly and update the existing document, does the document should be on the same index where current ingestion is happening or any old index of the same index pattern's doc can get updated?
i have configured rollover index in my case.

2

Hi @mathur7vidit

The output index must be the exact index ...
Not an index pattern

3

Thanks @stephenb . Is there a way to handle deleted docs then? it is still consuming disk space.. how can that be handled.

4

Deleted docs will be "merged" out eventually.

That process is a background task.

You should read this page carefully/ I'm detail.

5

Thanks @stephenb . i am aware of this force merge.. but reason i am asking about deleted docs is i have to keep only 1 index to manage the data which means it will always be a write index and not read-only. so how to handle deleted docs in this case.

6

They will get merged out eventually when it hits about 10% of the size or so I believe.

You can run the force merge with just only_expunge_deletes. Did you see that?

7

Hi @stephenb,

thanks for your response. i checked and verified that over a period of time that merging is happening and these deleted docs are gone. as overall docs count itself is not on the higher end, hence it is taking considerable time to perform merge activity.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.