How to speed up elasticsearch startup?

Elastic Stack Elasticsearch
1

Hi,

We are still using ES 5.1.2 in production as single instance.
Is there any way to speed up the startup time?

I restarted because I updated the used jre. Startup takes about 40 minutes until all indices are at status yellow.

top shows the following:

top - 12:22:54 up 95 days, 44 min,  3 users,  load average: 1.91, 2.20, 2.55
Tasks: 175 total,   1 running, 174 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us,  0.3 sy,  0.0 ni, 86.8 id, 12.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13186236+total,   617212 free, 16780212 used, 11446493+buff/cache
KiB Swap: 10484732 total, 10084992 free,   399740 used. 11034710+avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
27409 elastic+  20   0  0.574t 0.016t 1.430g S   6.3 13.0   7:13.68 java
   61 root      20   0       0      0      0 S   0.7  0.0  18:26.83 kswapd0
 4720 root      20   0   30084  10760   2736 S   0.3  0.0  64:14.28 filebeat
19035 logiweb+  20   0  159756   2308   1560 R   0.3  0.0   0:00.13 top
    1 root      20   0  195808   5696   3600 S   0.0  0.0  96:32.15 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:08.37 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   2:06.32 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      rt   0       0      0      0 S   0.0  0.0   4:38.58 migration/0
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      20   0       0      0      0 S   0.0  0.0 105:02.70 rcu_sched
   10 root      rt   0       0      0      0 S   0.0  0.0   3:20.08 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   2:27.83 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   4:00.98 migration/1

Wondering why the cpu load is that low and why io wait is only at 20%
8CPU, 128GB RAM.

Logstash and kibana are currently stopped. Nothing else running on the machine.

Thanks, Andreas

2

How many indices and shards do you have? How much data?

3

data size 627 GB
indices: 795 indices
shards: 3258

Is there a way to set an order in which the indices are loaded?
If so, I would like to use this:

.monitoring-* (xpack)
.kibana
all data indices starting with the newest following by older ones

4

That is a lot of quite small shards for a single node. How large portion of the indices are continuously being updated/added to?

5

indices are rotated each day.

a day is looking like this:

yellow open   .monitoring-es-2-2017.10.26      _PHbvUdTSLmeSPmGrl_t8A   1   1    3567205        12653        2gb            2gb
yellow open   other-prod-2017.10.26            5F5kSArZRSqa8kxfPRbMSw   1   1     283625            0    112.7mb        112.7mb
yellow open   perf-staging-2017.10.26          RjwLDHZ-Qee_gsqjlKkL4w   1   1     752292            0      266mb          266mb
yellow open   tux-prod-2017.10.26              knwJrZPhSyKJ0h0yiAe_Ew   1   1    8982863            0       10gb           10gb
yellow open   perf-prod-2017.10.26             wJ9P4iGfQ46-iBK9TdaDSA   1   1    1438301            0    754.4mb        754.4mb
yellow open   .monitoring-kibana-2-2017.10.26  jSLAIOHySnC6Ru_AIm6ioA   1   1       8608            0      2.1mb          2.1mb
yellow open   tux-staging-2017.10.26           OTwL64lsSeqxRz3qLdxUTw   1   1    1435865            0      1.2gb          1.2gb
yellow open   perf-no_stage_defined-2017.10.26 4uyTjp97RVm8Pgh2EgjEqA   5   1      39016            0     34.9mb         34.9mb
yellow open   other-staging-2017.10.26         ww_pp531TKyPEu1YAQ5EDg   1   1      17320            0        6mb            6mb

OK, I can reduce the shard size of perf-no_stage_defined, thought that was already done. But what can I do beside that?

We are only adding to the current day, maybe to yesterday if we had some problems.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.