Elasticsearch reindex is not working

Elastic Stack Elasticsearch
1

We have being trying to perform re indexing of existing to new index with change in mapping. But indexing is not happening properly.

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open ******* iso4UtTXSwuDTNrIMZXjpw 5 1 408486 84 10.4gb 5.2gb
green open ******* wSxaFkrXQJujiGOWhtyEDQ 5 1 408486 201678 29.6gb 15.8gb

docs.count is getting matched but not with docs.deleted and is not getting exited from sense plugin

2

What is the source? What is the target in your example?

3

green open AAAA iso4UtTXSwuDTNrIMZXjpw 5 1 408486 84 10.4gb 5.2gb
green open BBBB wSxaFkrXQJujiGOWhtyEDQ 5 1 408486 201678 29.6gb 15.8gb

we are re indexing AAAA to BBBB with new mapping

4

Issue in production.. We're not able re index it to new index with new mapping

5

So? What is the difference between old and new mapping?

Did you remove the BBBB index before launching the reindex operation?

6

We are adding date field on to the mapping and BBBB is new index
First we created the mapping and then tried reindexing the AAAA to BBBB

7

Can you reproduce with a simple example and share the full script so we can better understand?

8

David when we execute re indexing why do doc.deleted get increased? Is it normal behavior?

get /_cat/indices?v

9

That's what I want to understand actually. So that's the reason I asked for a full script which helps to understand what you are exactly doing. All the steps, please.

I suspect that you are doing updates of existing documents, not reindexing to a fresh and clean index. That's why I'm asking.

10

before this we create ticket_v2 mapping and run below command

POST _reindex
{
"source": {
"index": "ticket_v1"
},
"dest": {
"index": "ticket_v2"
}
}

11

So you don't delete ticket_v2?

12

Could you also post the exact command that you are using for the reindex? Feel free to black out the index name

Which version of Elasticsearch are you running? There have been a few bugs in the past

If you need a quick fix I would consider to expunge the deleted documents using the force merge API
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html
POST /twitter/_forcemerge?only_expunge_deletes=true

1 Like
13

We are not deleting ticket_v2. While re indexing to ticket_v2 doc.count matches with ticket_v1 but doc.deleted doesn't and count gets increased by 4 to 5 times. If you see first post wherein first line doc.deleted is 84 and in second line nearly 20 thousand.
What could be the issue?

14

why to do force merge it? we want to re index the data to new index with new mapping.

while performing the re indexing docs.count gets matched but only concern is docs.deleted number varies for long time . That is concern

we are using 5.2.0 version of elastic search.

POST _reindex
{
"source": {
"index": "ticket_v1"
},
"dest": {
"index": "ticket_v2"
}
}

15

So upgrade is the first thing to do.
Then create an empty target index. Which means DELETE target.
Then launch a single run of reindex from source to target and share the results here.

16

What makes to think we need upgrade?

17

This:

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.