Number of docs differents between primary shards and replicas

Elastic Stack Elasticsearch
1

hi everybody,
i have a 3 nodes 2.3.3 elk cluster.
i have one index named "idx_mys_pmm_iis-2017.02". 5 primary shards, 5 replicas shards.
Logstash 2.3.3 push the logs in this index.
in sense, i do :
"GET _cat/shards/idx_mys_pmm_iis-2017.02?v"

and i see this in response :
index shard prirep state docs store ip node
idx_mys_pmm_iis-2017.02 4 p STARTED 1809576 1.5gb node_UD0TX039
idx_mys_pmm_iis-2017.02 4 r STARTED 1809314 1.5gb node_UD0TX037
idx_mys_pmm_iis-2017.02 3 r STARTED 1808801 1.5gb node_UD0TX038
idx_mys_pmm_iis-2017.02 3 p STARTED 1808801 1.5gb node_UD0TX037
idx_mys_pmm_iis-2017.02 1 p STARTED 1811286 1.6gb node_UD0TX039
idx_mys_pmm_iis-2017.02 1 r STARTED 1811286 1.5gb node_UD0TX037
idx_mys_pmm_iis-2017.02 2 r STARTED 1808775 1.5gb node_UD0TX039
idx_mys_pmm_iis-2017.02 2 p STARTED 1808775 1.5gb node_UD0TX038
idx_mys_pmm_iis-2017.02 0 r STARTED 1809785 1.5gb node_UD0TX038
idx_mys_pmm_iis-2017.02 0 p STARTED 1809848 1.5gb node_UD0TX037

as u can see, it exists differences between primary and replicas shards for #0 and #4 shards.
then , I do a refresh on this index :
POST /idx_mys_pmm_iis-2017.02/_refresh

and this is what i get :
index shard prirep state docs store ip node
idx_mys_pmm_iis-2017.02 4 p STARTED 1811052 1.5gb node_UD0TX039
idx_mys_pmm_iis-2017.02 4 r STARTED 1811052 1.6gb node_UD0TX037
idx_mys_pmm_iis-2017.02 3 r STARTED 1810761 1.5gb node_UD0TX038
idx_mys_pmm_iis-2017.02 3 p STARTED 1810761 1.5gb node_UD0TX037
idx_mys_pmm_iis-2017.02 1 p STARTED 1813351 1.6gb node_UD0TX039
idx_mys_pmm_iis-2017.02 1 r STARTED 1813351 1.5gb node_UD0TX037
idx_mys_pmm_iis-2017.02 2 r STARTED 1810846 1.5gb node_UD0TX039
idx_mys_pmm_iis-2017.02 2 p STARTED 1810846 1.5gb node_UD0TX038
idx_mys_pmm_iis-2017.02 0 r STARTED 1811591 1.5gb node_UD0TX038
idx_mys_pmm_iis-2017.02 0 p STARTED 1811591 1.5gb node_UD0TX037

all is correct now...

can u explain to me why some events are in a primary shard and not in the replica's one (or vice versa) ?
and why, when i do a refresh, all become correct ... til the next refresh ...

ty very much for your answer...

2

The default for refresh should be 1s unless edited. What is your setting output for your index?

GET /idx_mys_pmm_iis-2017.02/_settings

3

hi Jimmy, ty for answering

this is the settings :
{
"idx_mys_pmm_iis-2017.02": {
"settings": {
"index": {
"creation_date": "1486557770568",
"refresh_interval": "-1",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "1QzkCRCJRDyjC58C_MUf1g",
"version": {
"created": "2030399"
}
}
}
}
}

4

You have disabled automatic refreshes, so there is no defined time limit for when all results will be made searchable. This mode is often used during bulk load, but requires a manual refresh or a change to the setting at the end. If you want to optimise for indexing performance, but do not have clearly defined bulk loads after which you can refresh, you can instead set this to 10 or 30 seconds and still see a performance benefit, while avoiding any manual steps being required.

5

ok ty christian.
i will change this refresh_interval setting.
can i say this :
the more the throuput of new documents is big, the more the value of the refresh_interval is high (ex: 10, 20 or 30 s)
but if you index a few documents by sec, u can set refresh_interval to 1s for example.
Am i wrong ?

6

If indexing is light, the default value of 1s should be fine.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.