Primary shard size much bigger than replica

saiergon · April 2, 2020, 11:33am

Elasticsearch version ( bin/elasticsearch --version ): 5.2.0

JVM version ( java -version ): 1.8.0_102

the primary shard size much bigger than the replica, like below, but the documents number is same：

use segment api , i found the primary shard only has one segment (because i issued a forcemerge operation), but the replica has 58 segments, the info as below:

"249" : [
          {
            "routing" : {
              "state" : "STARTED",
              "primary" : true,
              "node" : "pC6DBL4IQ6Ka3FCRj-bAoA"
            },
            "num_committed_segments" : 1,
            "num_search_segments" : 1,
            "segments" : {
              "_4gd" : {
                "generation" : 5773, 
                "num_docs" : 31580226,
                "deleted_docs" : 0,
                "size_in_bytes" : 52193301314,
                "memory_in_bytes" : 88592516,
                "committed" : true, 
                "search" : true, 
                "version" : "6.4.0",
                "compound" : false 
              }     
            }     
          },    
          {     
            "routing" : {
              "state" : "STARTED",
              "primary" : false,
              "node" : "5hK55XhzQNajssip8SUzRA"
            },    
            "num_committed_segments" : 58, 
            "num_search_segments" : 58, 
            "segments" : {
              "_ue" : {
                "generation" : 1094, 
                "num_docs" : 1391556,
                "deleted_docs" : 0,
                "size_in_bytes" : 2117655625,
                "memory_in_bytes" : 3299492,
                "committed" : true, 
                "search" : true, 
                "version" : "6.4.0",
                "compound" : false 
              },

and the node holds the primary shard hash much segments than the segment api returned.

total 99G
...
-rw-rw-r-- 1 search search  2.0G Apr  1 18:56 _3sy.cfs
-rw-rw-r-- 1 search search   405 Apr  1 18:56 _3sy.cfe
-rw-rw-r-- 1 search search   427 Apr  1 18:56 _3sy.si
-rw-rw-r-- 1 search search  2.0G Apr  1 19:32 _3w3.cfs
-rw-rw-r-- 1 search search   405 Apr  1 19:32 _3w3.cfe
-rw-rw-r-- 1 search search   427 Apr  1 19:32 _3w3.si
-rw-rw-r-- 1 search search  2.0G Apr  1 20:16 _3zs.cfs
-rw-rw-r-- 1 search search   405 Apr  1 20:16 _3zs.cfe
-rw-rw-r-- 1 search search   426 Apr  1 20:16 _3zs.si
-rw-rw-r-- 1 search search  2.0G Apr  1 20:46 _42j.cfs
-rw-rw-r-- 1 search search   405 Apr  1 20:46 _42j.cfe
-rw-rw-r-- 1 search search   427 Apr  1 20:46 _42j.si
-rw-rw-r-- 1 search search  2.0G Apr  1 21:24 _45k.cfs
-rw-rw-r-- 1 search search   405 Apr  1 21:24 _45k.cfe
-rw-rw-r-- 1 search search   427 Apr  1 21:24 _45k.si
-rw-rw-r-- 1 search search  2.0G Apr  1 22:14 _49i.cfs
-rw-rw-r-- 1 search search   405 Apr  1 22:14 _49i.cfe
-rw-rw-r-- 1 search search   427 Apr  1 22:14 _49i.si
-rw-rw-r-- 1 search search   389 Apr  1 22:51 _4ch.si
-rw-rw-r-- 1 search search   36M Apr  1 22:51 _4ch.cfs
-rw-rw-r-- 1 search search   405 Apr  1 22:51 _4ch.cfe
-rw-rw-r-- 1 search search   389 Apr  1 22:58 _4cw.si
-rw-rw-r-- 1 search search  4.1M Apr  1 22:58 _4cw.cfs
-rw-rw-r-- 1 search search   405 Apr  1 22:58 _4cw.cfe
-rw-rw-r-- 1 search search  2.0G Apr  1 23:02 _4cg.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:02 _4cg.cfe
-rw-rw-r-- 1 search search   426 Apr  1 23:02 _4cg.si
-rw-rw-r-- 1 search search   389 Apr  1 23:08 _4do.si
-rw-rw-r-- 1 search search  4.7M Apr  1 23:08 _4do.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:08 _4do.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:13 _4dz.si
-rw-rw-r-- 1 search search  5.4M Apr  1 23:13 _4dz.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:13 _4dz.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:13 _4e3.si
-rw-rw-r-- 1 search search  4.7M Apr  1 23:13 _4e3.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:13 _4e3.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:32 _4ez.si
-rw-rw-r-- 1 search search   61M Apr  1 23:32 _4ez.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:32 _4ez.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:32 _4f0.si
-rw-rw-r-- 1 search search  6.3M Apr  1 23:32 _4f0.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:32 _4f0.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:33 _4f3.si
-rw-rw-r-- 1 search search  2.9M Apr  1 23:33 _4f3.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:33 _4f3.cfe
-rw-rw-r-- 1 search search   14M Apr  1 23:33 _4f7.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:33 _4f7.cfe
-rw-rw-r-- 1 search search   427 Apr  1 23:33 _4f7.si
-rw-rw-r-- 1 search search   389 Apr  1 23:40 _4fc.si
-rw-rw-r-- 1 search search  4.7M Apr  1 23:40 _4fc.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:40 _4fc.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:40 _4fd.si
-rw-rw-r-- 1 search search  586K Apr  1 23:40 _4fd.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:40 _4fd.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:40 _4fg.si
-rw-rw-r-- 1 search search 1021K Apr  1 23:40 _4fg.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:40 _4fg.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:43 _4fi.si
-rw-rw-r-- 1 search search  7.2M Apr  1 23:43 _4fi.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:43 _4fi.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:43 _4fl.si
-rw-rw-r-- 1 search search  5.0M Apr  1 23:43 _4fl.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:43 _4fl.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:48 _4fq.si
-rw-rw-r-- 1 search search  1.9M Apr  1 23:48 _4fq.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:48 _4fq.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:48 _4fm.si
-rw-rw-r-- 1 search search   59M Apr  1 23:48 _4fm.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:48 _4fm.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:48 _4fs.si
-rw-rw-r-- 1 search search  794K Apr  1 23:48 _4fs.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:48 _4fs.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:48 _4fr.si
-rw-rw-r-- 1 search search  2.8M Apr  1 23:48 _4fr.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:48 _4fr.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:48 _4ft.si
-rw-rw-r-- 1 search search  3.4M Apr  1 23:48 _4ft.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:48 _4ft.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:53 _4fv.si
-rw-rw-r-- 1 search search   53M Apr  1 23:53 _4fv.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:53 _4fv.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:53 _4fw.si
-rw-rw-r-- 1 search search   21M Apr  1 23:53 _4fw.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:53 _4fw.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:53 _4fx.si
-rw-rw-r-- 1 search search   20M Apr  1 23:53 _4fx.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:53 _4fx.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:57 _4g1.si
-rw-rw-r-- 1 search search  3.3M Apr  1 23:57 _4g1.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:57 _4g1.cfe
-rw-rw-r-- 1 search search  1.9M Apr  1 23:57 _4g2.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:57 _4g2.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:57 _4g2.si
-rw-rw-r-- 1 search search   389 Apr  1 23:57 _4g4.si
-rw-rw-r-- 1 search search  496K Apr  1 23:57 _4g4.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:57 _4g4.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:57 _4g3.si
-rw-rw-r-- 1 search search  907K Apr  1 23:57 _4g3.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:57 _4g3.cfe
-rw-rw-r-- 1 search search   389 Apr  1 23:58 _4g7.si
-rw-rw-r-- 1 search search   27M Apr  1 23:58 _4g7.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:58 _4g7.cfe
-rw-rw-r-- 1 search search  280M Apr  1 23:59 _4g6.cfs
-rw-rw-r-- 1 search search   405 Apr  1 23:59 _4g6.cfe
-rw-rw-r-- 1 search search   427 Apr  1 23:59 _4g6.si
-rw-rw-r-- 1 search search  2.0G Apr  2 00:02 _4fu.cfs
-rw-rw-r-- 1 search search   405 Apr  2 00:02 _4fu.cfe
-rw-rw-r-- 1 search search   427 Apr  2 00:02 _4fu.si
-rw-rw-r-- 1 search search  103M Apr  2 00:03 _4gb.cfs
-rw-rw-r-- 1 search search   405 Apr  2 00:03 _4gb.cfe
-rw-rw-r-- 1 search search   427 Apr  2 00:03 _4gb.si
-rw-rw-r-- 1 search search  6.0M Apr  2 00:03 _4g9.cfs
-rw-rw-r-- 1 search search   405 Apr  2 00:03 _4g9.cfe
-rw-rw-r-- 1 search search   389 Apr  2 00:03 _4g9.si
-rw-rw-r-- 1 search search   389 Apr  2 00:03 _4g8.si
-rw-rw-r-- 1 search search   16M Apr  2 00:03 _4g8.cfs
-rw-rw-r-- 1 search search   405 Apr  2 00:03 _4g8.cfe
-rw-rw-r-- 1 search search   389 Apr  2 00:03 _4ga.si
-rw-rw-r-- 1 search search  113K Apr  2 00:03 _4ga.cfs
-rw-rw-r-- 1 search search   405 Apr  2 00:03 _4ga.cfe
-rw-rw-r-- 1 search search  3.8K Apr  2 00:04 segments_8d
-rw-rw-r-- 1 search search  7.1M Apr  2 03:01 _4gd.fdx
-rw-rw-r-- 1 search search   30G Apr  2 03:01 _4gd.fdt
-rw-rw-r-- 1 search search   72M Apr  2 03:15 _4gd_Lucene50_0.tip
-rw-rw-r-- 1 search search  8.2G Apr  2 03:15 _4gd_Lucene50_0.tim
-rw-rw-r-- 1 search search  881M Apr  2 03:15 _4gd_Lucene50_0.pos
-rw-rw-r-- 1 search search  1.2G Apr  2 03:15 _4gd_Lucene50_0.doc
-rw-rw-r-- 1 search search  2.4K Apr  2 03:23 _4gd_Lucene54_0.dvm
-rw-rw-r-- 1 search search  8.2G Apr  2 03:23 _4gd_Lucene54_0.dvd
-rw-rw-r-- 1 search search  500M Apr  2 03:24 _4gd.dim
-rw-rw-r-- 1 search search   122 Apr  2 03:24 _4gd.dii
-rw-rw-r-- 1 search search    78 Apr  2 03:24 _4gd.nvm
-rw-rw-r-- 1 search search   31M Apr  2 03:24 _4gd.nvd
-rw-rw-r-- 1 search search  5.1K Apr  2 03:24 _4gd.fnm
-rw-rw-r-- 1 search search   229 Apr  2 03:24 segments_8e
-rw-rw-r-- 1 search search   591 Apr  2 03:24 _4gd.si

DavidTurner · April 2, 2020, 11:40am

The older segments on the primary may be retained by ongoing activity (maybe searches or scrolls) and should eventually be cleaned up when that activity finishes. However this version is very old and long past the end of its supported life so maybe there's some behaviour that I don't know about which has been addressed in a more recent version too. You should definitely be trying to upgrade to a supported version as a matter of some urgency.

system · April 30, 2020, 11:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Primary and replica shard is very different Elasticsearch	2	578	July 27, 2018
Shard size is different between primary and replica Elasticsearch	3	1611	October 26, 2018
Different primary and replica shard size Elasticsearch	8	2246	July 29, 2019
Primary shard size higher than replicas Elasticsearch	1	432	December 8, 2020
Difference in shard size of primary and replica Elasticsearch	3	737	May 23, 2019

Primary shard size much bigger than replica

Related topics