I got much more sizes than base index after reindexation!

Elastic Stack Elasticsearch
1

Hello,

I wrote an application who does the reindexation. It reads from one index with scan and scroll and send it to another index. Here I got two things on the new index:

I have exactly the same size of the documents as the base index
I have %50 more size than the base index ( I have the same number of shards and replica)

Do you have any idea why I have more size while having the same number of documents and is this a problem?

EDIT: I looked at the segments settings the only difference is here

1---
segments: {
count: 97
memory_in_bytes: 42480824
index_writer_memory_in_bytes: 0
index_writer_max_memory_in_bytes: 2634022912
version_map_memory_in_bytes: 0
fixed_bit_set_memory_in_bytes: 0

2----

segments: {
count: 42
memory_in_bytes: 23748772
index_writer_memory_in_bytes: 0
index_writer_max_memory_in_bytes: 1024000
version_map_memory_in_bytes: 0
fixed_bit_set_memory_in_bytes: 0

Thank you

1 Like
2

How can you tell there is this difference?

Ultimately you'd have to compare the documents to see if there is a difference.

3

Have you changed any mappings between the two indices that could explain the size difference? Have you optimised any of the indices?

4

I am using the head plugin to visualize the indexes. So that I can see the number of documents and size.

5

If I am not wrong I did not use any optimize on the indexes but I will verify again.

6

Nope, I have done any optimization on the indexes.

7

How many segments? How many deleted docs? Those all have overhead.

8

Can you try to optimise both indices and see if the size difference changes?

9

I tried optimization too but no differences. Another question do we have to have exactly same size after reindexation ?

10

All of them identical

11

I have tried reindexation for many indexes. I got the same doc number every time, and nearly the same size for other indexes. But still got the difference for one of my indexes (%50 size difference), what can be the other possibilities ?

12

are u adding the content for the first time in that index? OR you are rewriting the same documents? What is the delete %age? If delete %age is different, try optimize (expunge delete) and compare sizes. It wont't be exact same size but not huge difference too.

13

Have you compared the mappings between these two indices? Are there any differences that could result in this size difference?