Records disappearing while using custom document_id

[karnamonkster (Karan)]

Hello,
We are trying to use a "Document_id" within ES output configuration (using multiple field columns from the Oracle Database table for example:

document_id => %{field1}%{field2}

Reason being to be able to have only one unique record in ES with the value field (field3) updated every minute with the latest near real-time value. (without creating duplicate records for each unique value)

However we noted that during a consequent run scheduled every minute using JDBC input, the record vanishes from the index if that document_id does not exists.

kindly help in this matter.

[warkolm (Mark Walkom)]

What index output pattern are you using?

[karnamonkster (Karan)]

There are no filters to my logstash configuration
Here is what it looks like

output { 
elasticsearch
{ 
hosts => ["array of hosts"]
document_id => "%{field1}%{field2}"
document_type => "flags"
index => "valflags-%{+YYYY.MM.dd}"
template => "/usr/share/...elasticsearch-template-es5x.json"
template_overwrite => true
}
}

[karnamonkster (Karan)]

Anything you seem is strange?