Records disappearing while using custom document_id

Hello,
We are trying to use a "Document_id" within ES output configuration (using multiple field columns from the Oracle Database table for example:

document_id => %{field1}%{field2}

Reason being to be able to have only one unique record in ES with the value field (field3) updated every minute with the latest near real-time value. (without creating duplicate records for each unique value)

However we noted that during a consequent run scheduled every minute using JDBC input, the record vanishes from the index if that document_id does not exists.

kindly help in this matter.

What index output pattern are you using?

There are no filters to my logstash configuration
Here is what it looks like

output { 
elasticsearch
{ 
hosts => ["array of hosts"]
document_id => "%{field1}%{field2}"
document_type => "flags"
index => "valflags-%{+YYYY.MM.dd}"
template => "/usr/share/...elasticsearch-template-es5x.json"
template_overwrite => true
}
}

Anything you seem is strange?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.