I got much more sizes than base index after reindexation!


(Mehmet Özer) #1

Hello,

I wrote an application who does the reindexation. It reads from one index with scan and scroll and send it to another index. Here I got two things on the new index:

I have exactly the same size of the documents as the base index
I have %50 more size than the base index ( I have the same number of shards and replica)

Do you have any idea why I have more size while having the same number of documents and is this a problem?

EDIT: I looked at the segments settings the only difference is here

1---
segments: {
count: 97
memory_in_bytes: 42480824
index_writer_memory_in_bytes: 0
index_writer_max_memory_in_bytes: 2634022912
version_map_memory_in_bytes: 0
fixed_bit_set_memory_in_bytes: 0

2----

segments: {
count: 42
memory_in_bytes: 23748772
index_writer_memory_in_bytes: 0
index_writer_max_memory_in_bytes: 1024000
version_map_memory_in_bytes: 0
fixed_bit_set_memory_in_bytes: 0

Thank you


(Mark Walkom) #2

How can you tell there is this difference?

Ultimately you'd have to compare the documents to see if there is a difference.


(Christian Dahlqvist) #3

Have you changed any mappings between the two indices that could explain the size difference? Have you optimised any of the indices?


(Mehmet Özer) #4

I am using the head plugin to visualize the indexes. So that I can see the number of documents and size.


(Mehmet Özer) #5

If I am not wrong I did not use any optimize on the indexes but I will verify again.


(Mehmet Özer) #6

Nope, I have done any optimization on the indexes.


(Nik Everett) #7

How many segments? How many deleted docs? Those all have overhead.


(Christian Dahlqvist) #8

Can you try to optimise both indices and see if the size difference changes?


(Mehmet Özer) #9

I tried optimization too but no differences. Another question do we have to have exactly same size after reindexation ?


(Mehmet Özer) #10

All of them identical


(Mehmet Özer) #11

I have tried reindexation for many indexes. I got the same doc number every time, and nearly the same size for other indexes. But still got the difference for one of my indexes (%50 size difference), what can be the other possibilities ?


(Imran Siddique) #12

are u adding the content for the first time in that index? OR you are rewriting the same documents? What is the delete %age? If delete %age is different, try optimize (expunge delete) and compare sizes. It wont't be exact same size but not huge difference too.


(Christian Dahlqvist) #13

Have you compared the mappings between these two indices? Are there any differences that could result in this size difference?


(system) #14