# Elasticsearch bulk index missing some records
**URL:** https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634
**Category:** Elasticsearch
**Created:** [July 5, 2018, 5:23am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634 "2018-07-05T05:23:58Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 5:23am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/1 "2018-07-05T05:23:58Z")
Hi,
i am using es 5.6 with python client and doing bulk index with some frequent log files.
what i need to care when going to bulk index and should not miss records. pls help
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 5:29am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/2 "2018-07-05T05:29:16Z")
Are you looking at the bulk response and checking that all documents were reported as successfully written?
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 5:36am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/3 "2018-07-05T05:36:33Z")
we are not looking for response, how can we handle it so successfully index all records.
any parameter or anything we should care, please suggest
our python client syntax : es = Elasticsearch([{'host': '192.168.1.xxx', 'port': 7205, 'timeout': 60},{'host': '192.168.1.yyy', 'port': 7205, 'timeout': 60}])
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 5:47am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/4 "2018-07-05T05:47:39Z")
You need to look at the bulk response and check that all records were successful. If some did fail, you need to handle those errors, e.g. by retrying them.
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 5:51am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/5 "2018-07-05T05:51:12Z")
What is possibility If we do not get error in response for record and that record not indexed in ES?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 5:52am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/6 "2018-07-05T05:52:27Z")
I do not think that should happen. If you get an acknowledgement without error from Elasticsearch the document has been indexed, although it may not yet be searchable unless a refresh has run.
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 7:10am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/7 "2018-07-05T07:10:33Z")
yes you are right, but we are processing a file having approx 500 records all doc indexed but few records missed, without any error, so what we should take care
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 7:15am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/8 "2018-07-05T07:15:53Z")
Are these documents reported as successfully indexed in the bulk response? How are you checking which documents are missing? have you run a refresh or waited for one to occur before checking if they have been indexed? Are you allowing Elasticsearch to assign document IDs?
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 8:58am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/9 "2018-07-05T08:58:18Z")
> [@Christian\_Dahlqvist](#):
>
> Are you allowing Elasticsearch to assign document IDs
we are having backup of processed files, in same file having 500 records few records are are not stored in es.
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 9:10am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/10 "2018-07-05T09:10:06Z")
we are giving our ID, oursample doc
{"index": {"\_index": "events", "\_type": "cm", "\_id": "1132052186974525\_65\_1"}}
{"DT": "2018-07-04T10:27:15", "RE": "1", "PT": "2018-07-05T09:13:26", "ZI": "1132052186974525", "RT": "65"}
{"index": {"\_index": "events", "\_type": "cm", "\_id": "1132052186974525\_65\_2"}}
{"DT": "2018-07-04T10:27:16", "RE": "2", "PT": "2018-07-05T09:13:26", "ZI": "1132052186974525", "RT": "65"}
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 5, 2018, 9:19am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/11 "2018-07-05T09:19:55Z")
> have you run a refresh or waited for one to occur before checking if they have been indexed?
Could you answer that?
Could you share the full output of the Bulk Response that your python job is getting?
If too big for this forum, upload as a [gist.github.com](http://gist.github.com) and share the link here.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 9:26am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/12 "2018-07-05T09:26:55Z")
Are you sure your document IDs are unique?
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 9:29am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/13 "2018-07-05T09:29:15Z")
yes doc is unique, default setting for refresh is applied which is false
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 9:29am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/14 "2018-07-05T09:29:57Z")
Default refresh time is 1 second. Is that what you are using?
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 11:11am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/15 "2018-07-05T11:11:09Z")
yes default is here
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 5, 2018, 11:16am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/16 "2018-07-05T11:16:04Z")
Do you have any non-default settings for Elasticsearch?
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 11:32am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/17 "2018-07-05T11:32:54Z")
no we have use default setting
---
### Author:  [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)
#### Post date: [July 5, 2018, 11:56am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/18 "2018-07-05T11:56:12Z")
I have observed following issues in bulk indexing -
- record that is included in bulk data post but not getting any response regarding that record
- So my conclusion is that somewhere in ES the recording is missing
I am giving few logs of ES - DEBUG mode
\_1. 2018-07-05 17:30:10,716 - root - DEBUG - File: /home/developer/p1\_py/logs/eventlogs/event2.log.2\_01 -- Response:{'errors': False, 'items': [ ..... {'index': {'\_shards': {'total': 2, 'successful': 2, 'failed': 0}, 'created': True, '\_index': 'campevents', '\_version': 1, 'result': 'created', '\_type': 'cmev', 'forced\_refresh': True, 'status': 201, '_id': '3160379186122643\_72\_2'}}_
\_2018-07-05 17:30:01,674 - root - DEBUG - Event data {"index": {"\_type": "cmev", "\_index": "campevents", "_id": "3160379186122643\_72\_2"}}:_
\_2. 2018-07-05 17:00:01,757 - root - DEBUG - Event data {"index": {"\_index": "campevents", "\_id": "1993761866142132\_77\_10", "_type": "cmev"}}:_
In above log we found that **"\_id": "1993761866142132\_77\_10"** only appeared in **Event data** log but didn't appeared in **Response:{'errors': False**
Please review the above logs and give us your opinion
\*\* we also change refresh interval 30 sec of that index and changed refresh : False, please suggest if missing anything so no chance to miss any data
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 2, 2018, 11:56am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-index-missing-some-records/138634/19 "2018-08-02T11:56:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.