Simple ingest pipeline reindex not getting all documents


(Norm Bedwell) #1

I am running a test on a simple reindex via a pipeline. I should be getting around 1500 documents according to the query, however, I am getting consistently around 900. I do see one or two failures but I seem to missing quite a few documents. Am I over looking something?

PUT _ingest/pipeline/process_history_reindex_pl
{
"description": "Pipeline to reindex process history prod index.",
"processors" : [
{
"json": {
"field": "historyRecord",
"target_field": "history"
}
}
],
"on_failure": [
{
"set": {
"field": "failed_id",
"value": "failed-{{_id}}"
}
}
]
}
POST _reindex
{
"source": {
"index": "bpo-processhistory-prod",
"query": {
"range": {
"time": {
"gte": "now-5m"
}
}
}
},
"dest": {
"index": "bpo-processhistory-prod-re",
"pipeline": "process_history_reindex_pl"
}
}


(Norm Bedwell) #2

Running an Elastic 5.5.0 cluster.

I have run some tests on this pipeline by changing the json processor to a rename. It will ingest all documents with the field renamed. It is only when the json processor is used that the input seems limited to around 900 documents.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.