# Node is getting hurt pretty bad from diagnostics information **URL:** \ **Category:** Elasticsearch\ **Created:** [September 16, 2013, 8:57pm UTC](https://discuss.elastic.co/t/node-is-getting-hurt-pretty-bad-from-diagnostics-information/13630 "2013-09-16T20:57:11Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Ashwin\_Sathya](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Ashwin\_Sathya](https://discuss.elastic.co/u/Ashwin_Sathya)\ **Post date:** [September 16, 2013, 8:57pm UTC](https://discuss.elastic.co/t/node-is-getting-hurt-pretty-bad-from-diagnostics-information/13630/1 "2013-09-16T20:57:11Z")
Hi, I have a 3 node cluster of XL VMs (14 Gig each, with 7G as ES\_MAX\_MEM) and i have pushed over 1 billion documents ( around 15 indices with 1 shard 2 replica each. Please bear with the sharding model ) around 300G of data. I noticed in HQ plugin that one of my machines was hurting bad. And decided to take a peek. The highlighted numbers are flagged as red in HQ. ES Uptime: 1.90 days 1.91 days 1.89 days CPU: Opteron Opteron Opteron # Cores: 8 8 8 Store Size: 111.4gb 105.2gb 105.2gb # Documents: 416440194 416651034 416651034 Documents Deleted: 0% 0% 0% Merge Size: 66.8gb 69.7gb 73.8gb Merge Time: 10.9d 3h 3.4h Merge Rate: 0.1 MB/s 6.7 MB/s 6.3 MB/s File Descriptors: -1 -1 -1 Index Activity Indexing - Index: 30.98ms 0.32ms 0.33ms Indexing - Delete: 0ms 0ms 0ms Search - Query: 5699.39ms 30.44ms 29.19ms Search - Fetch: 4996.11ms 16.99ms 15.04ms Get - Total: 0ms 0ms 0ms Get - Exists: 0ms 0ms 0ms Get - Missing: 0ms 0ms 0ms Refresh: 7366.72ms 64.32ms 71.41ms Flush: 108949.93ms 1484.93ms 1613.02ms Cache Activity Field Size: 0b 0b 0b Field Evictions: 0 0 0 Filter Cache Size: 25.7mb 29.1mb 30.3mb Filter Evictions: 0 per query 0 per query 0 per query ID Cache Size: 0b 0b 0b % ID Cache: 0% 0% 0% Memory Total Memory: 14 gb 14 gb 14 gb Heap Size: 2.8 gb 2.9 gb 2.9 gb Heap % of RAM: 20.2% 20.7% 20.4% % Heap Used: 66.2% 68.6% 59.2% GC MarkSweep Frequency: 671 s 965 s 786 s GC MarkSweep Duration: 970.69ms 39.81ms 36.12ms GC ParNew Frequency: 2 s 3 s 2 s GC ParNew Duration: 277.23ms 22.51ms 31.84ms Swap Space: 4915.43359375 mb 4781.4609375 mb 4710.9921875 mb Network HTTP Connection Rate: 0 /second 0 /second 0 /second I am also attaching a snippet of the log from the first machine in the list. Seems something weird. How do I make sense out of this ? What is going wrong ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [September 17, 2013, 4:37pm UTC](https://discuss.elastic.co/t/node-is-getting-hurt-pretty-bad-from-diagnostics-information/13630/2 "2013-09-17T16:37:20Z")
Hiya You have swap turned on, which is killing your machines during garbage collection. Turn off swap (either using bootstrap.mlockall or by just disabling swap entirely). Also, it looks like you are only querying the first node, which is probably why it has to do more garbage collection than the other two nodes. Clint On 16 September 2013 22:57, R Ashwin Sathya [ashwin.sathya@outlook.com](mailto:ashwin.sathya@outlook.com)wrote: > * * * > > Hi, > > I have a 3 node cluster of XL VMs (14 Gig each, with 7G as ES\_MAX\_MEM) and > i have pushed over 1 billion documents ( around 15 indices with 1 shard 2 > replica each. Please bear with the sharding model ) around 300G of data. I > noticed in HQ plugin that one of my machines was hurting bad. And decided > to take a peek.\*\*\*\* > > * * * > > The highlighted numbers are flagged as red in HQ.\*\*\*\* > > * * * > > _ES Uptime:_ \*\*\*\* > > 1.90 days \*\*\*\* > > 1.91 days \*\*\*\* > > 1.89 days \*\*\*\* > > _CPU:_ \*\*\*\* > > Opteron \*\*\*\* > > Opteron \*\*\*\* > > Opteron \*\*\*\* > > _# Cores:_ \*\*\*\* > > 8 \*\*\*\* > > 8 \*\*\*\* > > 8 \*\*\*\* > > _Store Size:_ \*\*\*\* > > 111.4gb \*\*\*\* > > 105.2gb \*\*\*\* > > 105.2gb \*\*\*\* > > _# Documents:_ \*\*\*\* > > 416440194 \*\*\*\* > > 416651034 \*\*\*\* > > 416651034 \*\*\*\* > > _Documents Deleted:_ \*\*\*\* > > 0% \*\*\*\* > > 0% \*\*\*\* > > 0% \*\*\*\* > > _Merge Size:_ \*\*\*\* > > 66.8gb \*\*\*\* > > 69.7gb \*\*\*\* > > 73.8gb \*\*\*\* > > _Merge Time:_ \*\*\*\* > > 10.9d \*\*\*\* > > 3h \*\*\*\* > > 3.4h \*\*\*\* > > _Merge Rate:_ \*\*\*\* > > 0.1 MB/s \*\*\*\* > > 6.7 MB/s \*\*\*\* > > 6.3 MB/s \*\*\*\* > > _File Descriptors:_ \*\*\*\* > > -1 \*\*\*\* > > -1 \*\*\*\* > > -1 \*\*\*\* > > _Index Activity_ > > _Indexing - Index:_ \*\*\*\* > > 30.98ms \*\*\*\* > > 0.32ms \*\*\*\* > > 0.33ms \*\*\*\* > > _Indexing - Delete:_ \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > _Search - Query:_ \*\*\*\* > > 5699.39ms \*\*\*\* > > 30.44ms \*\*\*\* > > 29.19ms \*\*\*\* > > _Search - Fetch:_ \*\*\*\* > > 4996.11ms \*\*\*\* > > 16.99ms \*\*\*\* > > 15.04ms \*\*\*\* > > _Get - Total:_ \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > _Get - Exists:_ \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > _Get - Missing:_ \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > 0ms \*\*\*\* > > _Refresh:_ \*\*\*\* > > 7366.72ms \*\*\*\* > > 64.32ms \*\*\*\* > > 71.41ms \*\*\*\* > > _Flush:_ \*\*\*\* > > 108949.93ms \*\*\*\* > > 1484.93ms \*\*\*\* > > 1613.02ms \*\*\*\* > > _Cache Activity_ > > _Field Size:_ \*\*\*\* > > 0b \*\*\*\* > > 0b \*\*\*\* > > 0b \*\*\*\* > > _Field Evictions:_ \*\*\*\* > > 0 \*\*\*\* > > 0 \*\*\*\* > > 0 \*\*\*\* > > _Filter Cache Size:_ \*\*\*\* > > 25.7mb \*\*\*\* > > 29.1mb \*\*\*\* > > 30.3mb \*\*\*\* > > _Filter Evictions:_ \*\*\*\* > > 0 per query \*\*\*\* > > 0 per query \*\*\*\* > > 0 per query \*\*\*\* > > _ID Cache Size:_ \*\*\*\* > > 0b \*\*\*\* > > 0b \*\*\*\* > > 0b \*\*\*\* > > _% ID Cache:_ \*\*\*\* > > 0% \*\*\*\* > > 0% \*\*\*\* > > 0% \*\*\*\* > > _Memory_ > > _Total Memory:_ \*\*\*\* > > 14 gb \*\*\*\* > > 14 gb \*\*\*\* > > 14 gb \*\*\*\* > > _Heap Size:_ \*\*\*\* > > 2.8 gb \*\*\*\* > > 2.9 gb \*\*\*\* > > 2.9 gb \*\*\*\* > > _Heap % of RAM:_ \*\*\*\* > > 20.2% \*\*\*\* > > 20.7% \*\*\*\* > > 20.4% \*\*\*\* > > _% Heap Used:_ \*\*\*\* > > 66.2% \*\*\*\* > > 68.6% \*\*\*\* > > 59.2% \*\*\*\* > > _GC MarkSweep Frequency:_ \*\*\*\* > > 671 s \*\*\*\* > > 965 s \*\*\*\* > > 786 s \*\*\*\* > > _GC MarkSweep Duration:_ \*\*\*\* > > 970.69ms \*\*\*\* > > 39.81ms \*\*\*\* > > 36.12ms \*\*\*\* > > _GC ParNew Frequency:_ \*\*\*\* > > 2 s \*\*\*\* > > 3 s \*\*\*\* > > 2 s \*\*\*\* > > _GC ParNew Duration:_ \*\*\*\* > > 277.23ms \*\*\*\* > > 22.51ms \*\*\*\* > > 31.84ms \*\*\*\* > > _Swap Space:_ \*\*\*\* > > 4915.43359375 mb \*\*\*\* > > 4781.4609375 mb \*\*\*\* > > 4710.9921875 mb \*\*\*\* > > _Network_ > > _HTTP Connection Rate:_ \*\*\*\* > > 0 /second \*\*\*\* > > 0 /second \*\*\*\* > > 0 /second\*\*\*\* > > * * * > > * * * > > I am also attaching a snippet of the log from the first machine in the > list. Seems something weird. \*\*\*\* > > * * * > > How do I make sense out of this ? What is going wrong ?\*\*\*\* > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ashwin\_Sathya](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@Ashwin\_Sathya](https://discuss.elastic.co/u/Ashwin_Sathya)\ **Post date:** [September 17, 2013, 6:14pm UTC](https://discuss.elastic.co/t/node-is-getting-hurt-pretty-bad-from-diagnostics-information/13630/3 "2013-09-17T18:14:20Z")
Thanks Clinton, I will tryout the swap space related issue. And you are correct from the fact that we are writing to one of the nodes always, (I run on windows, and every node has an endpoint and I am looking into alternate designs) but interestingly, the node which is spiked up is not the one whose endpoint I am directing my writes to. Seems strange. Any explanations ? Thanks, Ashwin Sathya From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Clinton Gormley Sent: September 17, 2013 22:07 To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Re: Node is getting hurt pretty bad from diagnostics information Hiya You have swap turned on, which is killing your machines during garbage collection. Turn off swap (either using bootstrap.mlockall or by just disabling swap entirely). Also, it looks like you are only querying the first node, which is probably why it has to do more garbage collection than the other two nodes. Clint On 16 September 2013 22:57, R Ashwin Sathya \<[ashwin.sathya@outlook.com](mailto:ashwin.sathya@outlook.com) [mailto:ashwin.sathya@outlook.com](mailto:ashwin.sathya@outlook.com) \> wrote: Hi, I have a 3 node cluster of XL VMs (14 Gig each, with 7G as ES\_MAX\_MEM) and i have pushed over 1 billion documents ( around 15 indices with 1 shard 2 replica each. Please bear with the sharding model ) around 300G of data. I noticed in HQ plugin that one of my machines was hurting bad. And decided to take a peek. The highlighted numbers are flagged as red in HQ. ES Uptime: 1.90 days 1.91 days 1.89 days CPU: Opteron Opteron Opteron # Cores: 8 8 8 Store Size: 111.4gb 105.2gb 105.2gb # Documents: 416440194 416651034 416651034 Documents Deleted: 0% 0% 0% Merge Size: 66.8gb 69.7gb 73.8gb Merge Time: 10.9d 3h 3.4h Merge Rate: 0.1 MB/s 6.7 MB/s 6.3 MB/s File Descriptors: -1 -1 -1 Index Activity Indexing - Index: 30.98ms 0.32ms 0.33ms Indexing - Delete: 0ms 0ms 0ms Search - Query: 5699.39ms 30.44ms 29.19ms Search - Fetch: 4996.11ms 16.99ms 15.04ms Get - Total: 0ms 0ms 0ms Get - Exists: 0ms 0ms 0ms Get - Missing: 0ms 0ms 0ms Refresh: 7366.72ms 64.32ms 71.41ms Flush: 108949.93ms 1484.93ms 1613.02ms Cache Activity Field Size: 0b 0b 0b Field Evictions: 0 0 0 Filter Cache Size: 25.7mb 29.1mb 30.3mb Filter Evictions: 0 per query 0 per query 0 per query ID Cache Size: 0b 0b 0b % ID Cache: 0% 0% 0% Memory Total Memory: 14 gb 14 gb 14 gb Heap Size: 2.8 gb 2.9 gb 2.9 gb Heap % of RAM: 20.2% 20.7% 20.4% % Heap Used: 66.2% 68.6% 59.2% GC MarkSweep Frequency: 671 s 965 s 786 s GC MarkSweep Duration: 970.69ms 39.81ms 36.12ms GC ParNew Frequency: 2 s 3 s 2 s GC ParNew Duration: 277.23ms 22.51ms 31.84ms Swap Space: 4915.43359375 mb 4781.4609375 mb 4710.9921875 mb Network HTTP Connection Rate: 0 /second 0 /second 0 /second I am also attaching a snippet of the log from the first machine in the list. Seems something weird. How do I make sense out of this ? What is going wrong ? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) . For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) . For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:15am UTC](https://discuss.elastic.co/t/node-is-getting-hurt-pretty-bad-from-diagnostics-information/13630/4 "2017-07-06T02:15:56Z")
Closed on July 6, 2017, 2:15am UTC