ElasticSearch index is not updating correctly

[avinash9999 (Avinash Tayal)]

I have a logstash code for updating elasticsearch index in every 5 minutes from my SQL server.
I am using action update to do this.

action => 'update'
There is a problem that my index is not updating with every records.
Few records and not getting updated.

 I am also using this same index in my python code for other tasks.
 Is it possible it can't update and search at same time causing miss-match in index data.??

Thanks in advance

[yaauie (Ry Biesemeyer)]

how is your jdbc input configured? By default, it attempts to select only new documents (see docs on sql_last_value).

[avinash9999 (Avinash Tayal)]

This is my jdbc input

jdbc {
jdbc_connection_string => "xxxxxxx"
jdbc_user => "xxxxx"
jdbc_password => "xxxxxxx"
jdbc_driver_library => "C:/sqljdbc_4.2/enu/jre8/sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
statement => "select
s.SurveyId,
s.LanguageId,
s.clientid,
s.SurveyStatusId,s.FirstCompleteDate
from
Surveys s (nolock)
where s.AccountId=1 and s.UpdateTimestamp > dateadd(hh,-4,:sql_last_value)"
last_run_metadata_path => "D:/Elasticsearch/logstashSurveys/logstash_last_run/sc-surveys.logstash_jdbc_last_run"
schedule => "*/20 * * * * *"
}

And This is my elasticsearch output

elasticsearch { 
  hosts => ["xxxx"]
  user => "xxx"
  password => "xxxx" 
  index => "sc-surveys"
  document_id => "%{surveyid}"
  action => 'update'
  doc_as_upsert => "true"
}