About indices stored size

Elastic Stack Elasticsearch
1

Hello,
My Logstash creates 17 indices daily basis, and stored size is as below.

Should I reduce Shard number for small sized indices?
OR merge some of indices?
e.g. combine "access_xpg_prd_20180928", "access_ui50_prd_20180928" and "access_scs_prd_20180928" to "access_prd_20180928"

GET _cat/indices/*20180928?v&s=index

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open access_metv_prd_20180928 eGUZymINRkG4jaMzUJMmhA 6 1 1028172 0 1.5gb 778.4mb
green open access_ncms_prd_20180928 nrP-FZIYSi2E_9Rw-KA6xA 6 1 60593 0 62.2mb 30.9mb
green open access_scs_prd_20180928 r2Wzgv4zS2Kszdr-o4XOzQ 6 1 186080 0 331.4mb 165.6mb
green open access_ui50_prd_20180928 FbGCqHQgQ2ypYtSZKADdLA 6 1 25823 0 19.8mb 9.9mb
green open access_xpg_prd_20180928 rQsZBxvWTUOCLVE9e1EunQ 6 1 1457313 0 1.8gb 959.1mb
green open gateway_prd_20180928 u-cFpU3HRL6DSRx-hk_q-Q 6 1 16299200 0 35.5gb 16.9gb
green open metv_prd_metv-bookmark_20180928 -b7T9yskSDm0uoD6b-fevQ 6 1 1023104 0 936.8mb 468.3mb
green open metv_prd_metv-datamart_20180928 tHungQ4_QCa35oSKZ3jGoA 6 1 3562219 0 2.5gb 1.2gb
green open metv_prd_metv-purchase_20180928 GsbXY3q7QAyRyK-UPcN18A 6 1 135541 0 102.2mb 51.2mb
green open metv_prd_metv-setting_20180928 bJ7u45paSDqArEOHozuD2w 6 1 2495677 0 1.9gb 976.6mb
green open metv_prd_metv-watch_20180928 TCq8opsGToOYNy3-HRp6HA 6 1 5490844 0 4.8gb 2.3gb
green open ncms_prd_ncms-api_20180928 ALaztRz9Q7ONtyIheVK9ug 6 1 4692 0 3.1mb 1.5mb
green open ncms_prd_ncms-app_20180928 5-AcBiwpSoWcFJ4aA2FrYw 6 1 303754 0 211.7mb 105.9mb
green open scs_prd_scs-gwsvc_20180928 1y5RfTixSP-WVwA0aX0B7A 6 1 705293 0 1.4gb 738.8mb
green open scs_prd_scs-purchase_20180928 Z5uPSEiITRiwAQpI3VP4Og 6 1 645598 0 1gb 519.4mb
green open scs_prd_scs-stbauth_20180928 JdIzJvDvSA62ntHmbLQUag 6 1 2454 0 4.9mb 2.5mb
green open scs_prd_scs-triggersvc_20180928 jFXjcOd1RmyQoIiSWUXmZw 6 1 46477 0 20.5mb 10.2mb

2

Having lots of small shards like this can be very inefficient. All your shards are very small, so I do not see any need for more than a single primary shard for any of them. Merging indices is also an option. If you have a long retention period I would even recommend switching to monthly indices for the vast majority of them, even after they are combined.

3

That was what i wated to heard from you to confirm. Yes we combined some of similar indices today.

Can you please advise or share the reference page how i can combine/merge old indices in warm nodes??

Thank you!!

4

You will need to use the reindex API.

5

Okay I will take a look. Thank you!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.