Reindexing not copying all data

[alajangi (Kishore)]

Hi,

my elasticsearch version is 6.5.1

I want to clone an index, I am doing it in kibana devtools console,

POST _reindex
{
  "source": {
    "index": "source"
  },
  "dest": {
    "index": "dest"
      }
}

I am unable to get whole data, just getting 997 docs out of 157k docs, what am i doing wrong? also the destination index always up to date whenever the source index updated, could someone assist me ?

Thank you,
Kishore Kumar.

[Rahul_Kumar4 (Rahul Kumar)]

One reason why this could be happening is because your source index may have duplicate docs. Try a UNIQUE COUNT aggregation on the field _id in Kibana visualization on your source index and see what number this is.

If you want to do a full copy try setting the op_type in destination part of your request body to create. The default is index which just updates the doc if it already existed. - https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

[alajangi (Kishore)]

Hi Rahul,
I reindexed again by using op_type, and checked "unique count" in kibana both source and dest indexes, source: 124 and dest: 997, how is it possible? when i am trying to access new index in my python program i am getting "RequestError(400, 'search_phase_execution_exception', 'Fielddata is disabled on text fields by default. Set fielddata=true on [job] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.')" error, i am able to access without any issue with old index though,
Thanks,
Kishore.

[alajangi (Kishore)]

Hi,

I did mapping same as source index before reindex, but the UNIQUE COUNT in kibana visualize showing huge difference, source is 230 dest is 138,884, anyway the documents are same 158k in both, also i am able to access the index from my python program.
can anyone help me how to keep update my new index with my source index, say if any new data come to my source index, it has to update on my new index also?

Thank you,
Kishore.