# Simple ingest pipeline reindex not getting all documents

**URL:** <https://discuss.elastic.co/t/simple-ingest-pipeline-reindex-not-getting-all-documents/137783>\
**Category:** Elasticsearch\
**Created:** [June 28, 2018, 12:04pm UTC](https://discuss.elastic.co/t/simple-ingest-pipeline-reindex-not-getting-all-documents/137783 "2018-06-28T12:04:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [June 28, 2018, 12:04pm UTC](https://discuss.elastic.co/t/simple-ingest-pipeline-reindex-not-getting-all-documents/137783/1 "2018-06-28T12:04:15Z")

</div>

I am running a test on a simple reindex via a pipeline. I should be getting around 1500 documents according to the query, however, I am getting consistently around 900. I do see one or two failures but I seem to missing quite a few documents. Am I over looking something?

PUT \_ingest/pipeline/process\_history\_reindex\_pl  
{  
"description": "Pipeline to reindex process history prod index.",  
"processors" : [  
{  
"json": {  
"field": "historyRecord",  
"target\_field": "history"  
}  
}  
],  
"on\_failure": [  
{  
"set": {  
"field": "failed\_id",  
"value": "failed-{{\_id}}"  
}  
}  
]  
}  
POST \_reindex  
{  
"source": {  
"index": "bpo-processhistory-prod",  
"query": {  
"range": {  
"time": {  
"gte": "now-5m"  
}  
}  
}  
},  
"dest": {  
"index": "bpo-processhistory-prod-re",  
"pipeline": "process\_history\_reindex\_pl"  
}  
}

---

<div class="post-metadata">

**Author:** ![chapmantrain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chapmantrain/32/22646_2.png) [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\
**Post date:** [June 28, 2018, 2:19pm UTC](https://discuss.elastic.co/t/simple-ingest-pipeline-reindex-not-getting-all-documents/137783/2 "2018-06-28T14:19:04Z")

</div>

Running an Elastic 5.5.0 cluster.

I have run some tests on this pipeline by changing the json processor to a rename. It will ingest all documents with the field renamed. It is only when the json processor is used that the input seems limited to around 900 documents.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 26, 2018, 2:19pm UTC](https://discuss.elastic.co/t/simple-ingest-pipeline-reindex-not-getting-all-documents/137783/3 "2018-07-26T14:19:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.