# Elasticsearch losing data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 4, 2016, 3:18pm UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953 "2016-12-04T15:18:00Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![keshav](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@keshav](https://discuss.elastic.co/u/keshav)\ **Post date:** [December 4, 2016, 3:18pm UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/1 "2016-12-04T15:18:01Z")
I have a single node ES cluster with 26 GB RAM, 4 CPUs and 13 GB Heap Space. I am loading data into ES using threading in bulks of 5000 documents, the number of threads running can vary from 5 to 10. The problem that I am facing is that I am loading a `true/false` flag which is being lost by ES. Every data field exists in the final documents except that particular flag. Some documents contain the flag and some don't. The ratio of exist to not exist is about 2:7. The piece of code that introduces that flag is a fairly simple if-else block `if foo: doc['flag'] = True else: doc['flag'] = False` I have created a mapping for the index and specifically set the field type as `boolean`; I am unable to find a reason for the loss of the field. Any help would be greatly appreciated. Note:- I am using ES 2.3.3 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 4, 2016, 4:44pm UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/2 "2016-12-04T16:44:41Z")
Do you mean that \_source document does not contain the Boolean but other fields? If so you have an issue in your injector. ---
**Author:** ![keshav](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@keshav](https://discuss.elastic.co/u/keshav)\ **Post date:** [December 4, 2016, 5:07pm UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/3 "2016-12-04T17:07:29Z")
Yes, that is the problem. I have checked the injector code and have checked for the existence of the flag in list of fields in every document, which returns that the flag exists; but when they are sent over to ES, the flag doesn't exist in `_source`. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 4, 2016, 5:30pm UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/4 "2016-12-04T17:30:30Z")
This can not happen on elasticsearch side. May be you are doing updates and not inserts using bulk but you don't control the response? Anyhow you need to provide more details and may be your injector code. ---
**Author:** ![keshav](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@keshav](https://discuss.elastic.co/u/keshav)\ **Post date:** [December 5, 2016, 5:29am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/5 "2016-12-05T05:29:56Z")
I am overwriting the existing documents. I'm using python to inject data into ES, this is my injector code: ``` import csv import json from elasticsearch import Elasticsearch, helpers client = Elasticsearch("http://localhost:9200") entries = csv.DictReader(open("~/entries.csv")) images_ids = json.load(open("~/image_ids.json")) to_send = set() for entry in entries: if entry['image_id'] in images_ids: entry['duplicate'] = True else: entry['duplicate'] = False entry = { "_index": "my_index", "_type": "entries", "_id": entry['id'], '_source': entry } to_send.add(entry) if len(to_send) >= 5000: helpers.bulk(client, to_send) to_send = set() ``` As you can see, it is a simple if else block that is working on the duplicate flag. Is this the correct way or should I use another method to index the documents? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 5, 2016, 5:47am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/6 "2016-12-05T05:47:38Z")
Do you read the bulk response? ---
**Author:** ![keshav](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@keshav](https://discuss.elastic.co/u/keshav)\ **Post date:** [December 5, 2016, 6:12am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/7 "2016-12-05T06:12:28Z")
I tried printing it and it prints `(5000, [])` I'm assuming that should be number of products indexed and errors. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 5, 2016, 6:41am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/8 "2016-12-05T06:41:48Z")
Is it failing always for the same doc at every run? Can you reproduce it? Ideally with a REST script? ---
**Author:** ![keshav](https://avatars.discourse-cdn.com/v4/letter/k/b5e925/32.png) [@keshav](https://discuss.elastic.co/u/keshav)\ **Post date:** [December 5, 2016, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/9 "2016-12-05T07:32:40Z")
No, it fails randomly for any number of docs. By reproducing do you mean that I should try indexing the documents using the bulk api in ES using REST instead of the method I currently use? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 5, 2016, 8:00am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/10 "2016-12-05T08:00:48Z")
I was meaning something we can use to reproduce on our end. If it fails randomly, can you print again the result of the bulk after it has failed? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 2, 2017, 8:00am UTC](https://discuss.elastic.co/t/elasticsearch-losing-data/67953/11 "2017-01-02T08:00:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.