# ElasticSearch High Search response times on new Machine **URL:** \ **Category:** Elasticsearch\ **Created:** [April 26, 2022, 1:53pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293 "2022-04-26T13:53:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dnsh](https://avatars.discourse-cdn.com/v4/letter/d/ecc23a/32.png) [@dnsh](https://discuss.elastic.co/u/dnsh)\ **Post date:** [April 26, 2022, 1:53pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293/1 "2022-04-26T13:53:27Z")
I have 4 EC2 machines in Elasticsearch Cluster. Configuration: `c5d.large`, Memory: `3.5G`, Data Disk: `50GB NVME` instance storage. Elasticsearch Version: `6.8.21` I added the 5th machine with the same configuration `c5d.large`, Memory: `3.5G`, Data Disk: `50GB NVME` instance storage. After that, Search requests are taking more time than earlier. I enabled slow logs, which shows only shards that are present on the 5th node are taking more time for search. Also, I can see high disk Read IO happening on new node when I trigger search requests. The iowait% increases by the number of search requests and goes up to 90-95%. All old nodes do not show any read spikes. I checked `elasticsearch.yml`, `jvm.options` and even `sysctl -A` configurations. there is no diff between config on new nodes vs old nodes. What could be the issue here? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 26, 2022, 2:16pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293/2 "2022-04-26T14:16:49Z")
Did all the shard shuffling complete? Were the searches performed during the rebalance? BTW 6.8 is EOL you should consider upgrading as a matter of urgency. ---
**Author:** ![dnsh](https://avatars.discourse-cdn.com/v4/letter/d/ecc23a/32.png) [@dnsh](https://discuss.elastic.co/u/dnsh)\ **Post date:** [April 26, 2022, 3:03pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293/3 "2022-04-26T15:03:36Z")
Shard shuffling was already completed. I also waited for 20 minutes for the CPU to stabilize. then, I triggered search requests. Also, Disk Reads spikes only when I trigger search requests and only on new machine. Few Details about index: ```auto { "health": "green", "status": "open", "index": "my-index-name", "uuid": "m1ZlUa5USgKAUy1MkSasJg", "pri": "12", "rep": "1", "docs.count": "64542359", "docs.deleted": "19423498", "store.size": "83gb", "pri.store.size": "40gb" } ``` Regarding EOL, Yes, We are planning to move to the latest version of ES after few months. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [April 26, 2022, 3:12pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293/4 "2022-04-26T15:12:32Z")
> [@dnsh](#): > > ```auto > "health": "green", > "status": "open", > "index": "my-index-name", > "uuid": "m1ZlUa5USgKAUy1MkSasJg", > "pri": "12", > "rep": "1", > "docs.count": "64542359", > "docs.deleted": "19423498", > > ``` Not sure... you could try force merge with expunge delete to clean up the segments. Perhaps on the other nodes the data is already cached in RAM... Otherwise perhaps something is different with the underlying HW / OS. ---
**Author:** ![dnsh](https://avatars.discourse-cdn.com/v4/letter/d/ecc23a/32.png) [@dnsh](https://discuss.elastic.co/u/dnsh)\ **Post date:** [April 28, 2022, 7:37pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293/5 "2022-04-28T19:37:42Z")
I tried with a new VM. Getting the same issue. So there is nothing wrong with the provisioned VM. Well, I found diff in `lscpu` in old vs new vms. New VM has better CPU. Please find below lscpu output. Also, new VM has one extra cpu flag - `invpcid_single` and it does not have `hle` and `rtm` flags compred to old VMs. **New VM** ```auto lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Stepping: 7 CPU MHz: 2999.998 BogoMIPS: 5999.99 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 36608K NUMA node0 CPU(s): 0,1 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke ``` **Old VM** ```auto lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 2 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz Stepping: 4 CPU MHz: 3000.000 BogoMIPS: 6000.00 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0,1 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 ida arat pku ospke ``` * * * * * * Regarding this comment, > Perhaps on the other nodes the data is already cached in RAM... This is not the case. I restarted Elasticsearch process on old VMs. Even after the restart, I do not see any disk spikes. Does Elasticsearch maintains some kind of computations/cache on disk? * * * * * * ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 26, 2022, 7:38pm UTC](https://discuss.elastic.co/t/elasticsearch-high-search-response-times-on-new-machine/303293/6 "2022-05-26T19:38:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.