# The records in the elastic index are missing frequenlty
**URL:** https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738
**Category:** Logstash
**Created:** [April 13, 2020, 6:50am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738 "2020-04-13T06:50:14Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 13, 2020, 6:50am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/1 "2020-04-13T06:50:14Z")
I had a record in the elastic index which was indexed by logstash from a sql data source. This record which is already came to elastic index is not available after sometime. After updating the record in the database does the logstash removes the existed record in the elastic index and it will re-index the record with the new updated record ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 13, 2020, 7:46am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/2 "2020-04-13T07:46:54Z")
What does your Logstash config look like?
---
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 13, 2020, 9:16am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/3 "2020-04-13T09:16:05Z")
it is a normal config file that indexes the sql table data. it contains input jdbc plugin,filter json, output es
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 13, 2020, 9:20am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/4 "2020-04-13T09:20:10Z")
Without seeing the file, at least elasticsearch output and sections setting fields used in this, I can not do much to help.
How are you establishing that documents are missing?
---
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 13, 2020, 9:37am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/5 "2020-04-13T09:37:35Z")
Here already the document is indexed after some time updating the document, it is not available in elastic index. how once indexed document suddenly disappers.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 13, 2020, 9:43am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/6 "2020-04-13T09:43:58Z")
How are you querying for the document?
Are there any errors in the Logstash or Elasticsearch logs?
What does the relevant parts of your Logstash config look like?
---
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 13, 2020, 9:56am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/7 "2020-04-13T09:56:06Z")
```auto
input {
jdbc {
jdbc_connection_string =>""
jdbc_user =>""
jdbc_password =>""
jdbc_driver_class =>"com.mysql.jdbc.Driver"
schedule =>"* * * * *"
statement =>"SELECT * FROM blog_pro WHERE last_modified_time >:sql_last_value"
use_column_value =>true
tracking_column =>last_modified_time
tracking_column_type => "timestamp"
}
}
filter{
json {
source => "post"
target => "Post"
remove_field => ["post"]
}
output
{
#output to elasticsearch
elasticsearch {
hosts => ["localhost:9200"]
index => "idx_index"
document_id => "%{id}"
}
}
```
it looks like this
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 13, 2020, 1:07pm UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/8 "2020-04-13T13:07:41Z")
What about the other questions I had?
---
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 14, 2020, 1:29am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/9 "2020-04-14T01:29:21Z")
1. I am querying the document with the document id in the elastic head
2. There are no errors in elastic & logstash logs
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 14, 2020, 5:51am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/10 "2020-04-14T05:51:17Z")
Logstash does not delete anything. If a new document with the same ID as an existing document comes in it will simply overwrite that document. I can therefore not see any reason why a document would vanish. If something failed in the update I would expect you to continue seeing the old version.
Which version of Logstash and Elasticsearch are you using? How many documents does your index contain? How many nodes do you have in the cluster?
---
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 14, 2020, 6:10am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/11 "2020-04-14T06:10:59Z")
logstash version 6.4.2 and elastic version 6.4.2
I'm indexing 8000 documents and i had only one node in cluster
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 14, 2020, 6:49am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/12 "2020-04-14T06:49:28Z")
What does the output of the [cat indices API](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/cat-indices.html) look like? What does the query you are running to find the document look like?
---
### Author:  [@tharak](https://discuss.elastic.co/u/tharak)
#### Post date: [April 14, 2020, 7:25am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/13 "2020-04-14T07:25:25Z")
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open idx\_index nYFWZEO7TUiOjLQXBaYJpA 6474 0 0 0 260mb 260mb
query: {
```
bool: {
must: [
{
term: {
"Id.keyword": "1453"
}
},
{
term: {
"IsActive": true,
}
}
]
}
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 14, 2020, 7:32am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/14 "2020-04-14T07:32:55Z")
What do you get if you just run `GET idx_index/doc/1453` (Might be `_doc` instead of `doc` depending on version used..)?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 12, 2020, 7:43am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738/15 "2020-05-12T07:43:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.