# Elastic Search deleting some files while indexing? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 7, 2014, 2:58pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671 "2014-02-07T14:58:11Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [February 7, 2014, 2:58pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/1 "2014-02-07T14:58:11Z")
I am indexing about 5000 documents, when indexing is done, I use "HEAD" plugin, it says it indexed 4950 docs and deleted 50 files, also verified by curl that only 4950 indexed. I couldn't see anything in the logs, but how/when/why does Elasticsearch decide to delete some of the docs? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2014, 3:27pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/2 "2014-02-07T15:27:24Z")
May be you updated 50 docs (same ID)? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 7 février 2014 at 15:58:15, ZenMaster80 ([sabdalla80@gmail.com](mailto:sabdalla80@gmail.com)) a écrit: ## I am indexing about 5000 documents, when indexing is done, I use "HEAD" plugin, it says it indexed 4950 docs and deleted 50 files, also verified by curl that only 4950 indexed. I couldn't see anything in the logs, but how/when/why does Elasticsearch decide to delete some of the docs? You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52f4fb5c.74b0dc51.12f3e%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52f4fb5c.74b0dc51.12f3e%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [February 7, 2014, 4:51pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/3 "2014-02-07T16:51:23Z")
No, because it is a totally new index. I tried it several times, deleted the index, then created/indexed. On Friday, February 7, 2014 10:27:24 AM UTC-5, David Pilato wrote: > May be you updated 50 docs (same ID)? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > Le 7 février 2014 at 15:58:15, ZenMaster80 ([sabda...@gmail.com](mailto:sabda...@gmail.com)\) > a écrit: > > ## I am indexing about 5000 documents, when indexing is done, I use "HEAD" plugin, it says it indexed 4950 docs and deleted 50 files, also verified by curl that only 4950 indexed. I couldn't see anything in the logs, but how/when/why does Elasticsearch decide to delete some of the docs? > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2014, 5:23pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/4 "2014-02-07T17:23:01Z")
I understood that's a new index. It does not mean that in your insert script you don't have duplicate IDs. I would first check my documents. BTW elasticsearch does not delete documents unless you set \_ttl. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 7 février 2014 at 17:51:28, ZenMaster80 ([sabdalla80@gmail.com](mailto:sabdalla80@gmail.com)) a écrit: No, because it is a totally new index. I tried it several times, deleted the index, then created/indexed. On Friday, February 7, 2014 10:27:24 AM UTC-5, David Pilato wrote: May be you updated 50 docs (same ID)? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 7 février 2014 at 15:58:15, ZenMaster80 ([sabda...@gmail.com](mailto:sabda...@gmail.com)) a écrit: ## I am indexing about 5000 documents, when indexing is done, I use "HEAD" plugin, it says it indexed 4950 docs and deleted 50 files, also verified by curl that only 4950 indexed. I couldn't see anything in the logs, but how/when/why does Elasticsearch decide to delete some of the docs? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52f51675.2eb141f2.12f3e%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52f51675.2eb141f2.12f3e%40MacBook-Air-de-David.local). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [February 7, 2014, 5:32pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/5 "2014-02-07T17:32:54Z")
Sorry, I misunderstood the first time. It makes sense, I will take a look. On Friday, February 7, 2014 12:23:01 PM UTC-5, David Pilato wrote: > I understood that's a new index. > It does not mean that in your insert script you don't have duplicate IDs. > > I would first check my documents. > > BTW elasticsearch does not delete documents unless you set \_ttl. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > Le 7 février 2014 at 17:51:28, ZenMaster80 ([sabda...@gmail.com](mailto:sabda...@gmail.com)\) > a écrit: > > No, because it is a totally new index. I tried it several times, deleted > the index, then created/indexed. > > On Friday, February 7, 2014 10:27:24 AM UTC-5, David Pilato wrote: > > > May be you updated 50 docs (same ID)? > > > > ``` > > -- > > > > ``` > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > > Le 7 février 2014 at 15:58:15, ZenMaster80 ([sabda...@gmail.com](mailto:sabda...@gmail.com)) a écrit: > > > > ## I am indexing about 5000 documents, when indexing is done, I use "HEAD" plugin, it says it indexed 4950 docs and deleted 50 files, also verified by curl that only 4950 indexed. I couldn't see anything in the logs, but how/when/why does Elasticsearch decide to delete some of the docs? > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/07e4262e-bfc2-4417-b6b3-d42123da46b5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/07e4262e-bfc2-4417-b6b3-d42123da46b5%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![IronMike](https://avatars.discourse-cdn.com/v4/letter/i/e9a140/32.png) [@IronMike](https://discuss.elastic.co/u/IronMike)\ **Post date:** [February 7, 2014, 6:01pm UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/6 "2014-02-07T18:01:57Z")
Thanks David, I had duplicates like you mentioned. On Friday, February 7, 2014 12:23:01 PM UTC-5, David Pilato wrote: > I understood that's a new index. > It does not mean that in your insert script you don't have duplicate IDs. > > I would first check my documents. > > BTW elasticsearch does not delete documents unless you set \_ttl. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > Le 7 février 2014 at 17:51:28, ZenMaster80 ([sabda...@gmail.com](mailto:sabda...@gmail.com)\) > a écrit: > > No, because it is a totally new index. I tried it several times, deleted > the index, then created/indexed. > > On Friday, February 7, 2014 10:27:24 AM UTC-5, David Pilato wrote: > > > May be you updated 50 docs (same ID)? > > > > ``` > > -- > > > > ``` > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > > Le 7 février 2014 at 15:58:15, ZenMaster80 ([sabda...@gmail.com](mailto:sabda...@gmail.com)) a écrit: > > > > ## I am indexing about 5000 documents, when indexing is done, I use "HEAD" plugin, it says it indexed 4950 docs and deleted 50 files, also verified by curl that only 4950 indexed. I couldn't see anything in the logs, but how/when/why does Elasticsearch decide to delete some of the docs? > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0e4974a-9add-42bc-a0c3-e2aefbae2441%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ada899f6-c035-4369-8f38-8b436f6375fa%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ccf35bb0-e282-4b39-a6f9-5aabf34f9e1f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ccf35bb0-e282-4b39-a6f9-5aabf34f9e1f%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:51am UTC](https://discuss.elastic.co/t/elastic-search-deleting-some-files-while-indexing/15671/7 "2017-07-06T01:51:42Z")