# Encountered monitor.jvm warning **URL:** \ **Category:** Elasticsearch\ **Created:** [March 24, 2015, 9:26am UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848 "2015-03-24T09:26:21Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Abid\_Hussain](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@Abid\_Hussain](https://discuss.elastic.co/u/Abid_Hussain)\ **Post date:** [March 24, 2015, 9:26am UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848/1 "2015-03-24T09:26:21Z")
Hi all, we today got (for the first time) warning messages which seem to indicate a memory problem: [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total [5m]/[16.7m], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] [6.9gb]-\>[3.7gb]/[8.5gb]} [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total [18.4s]/[58s], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] [6.9gb]-\>[3.7gb]/[8.5gb]} [2015-03-24 09:08:15,372][WARN][monitor.jvm] [Danger] [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], total [1.4s]/[16.7m], memory [3.7gb]-\>[5gb]/[9.8gb], all\_pools {[young] [6.1mb]-\>[2.7mb]/[1.1gb]}{[survivor] [0b]-\>[149.7mb]/[149.7mb]}{[old] [3.7gb]-\>[4.9gb]/[8.5gb]} [2015-03-24 09:08:18,192][WARN][monitor.jvm] [Danger] [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], total [1.4s]/[16.7m], memory [5.8gb]-\>[6.2gb]/[9.8gb], all\_pools {[young] [845.4mb]-\>[1.2mb]/[1.1gb]}{[survivor] [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] [4.9gb]-\>[6gb]/[8.5gb]} [2015-03-24 09:08:21,506][WARN][monitor.jvm] [Danger] [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], total [1.2s]/[16.7m], memory [7gb]-\>[7.3gb]/[9.8gb], all\_pools {[young] [848.6mb]-\>[2.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] [6gb]-\>[7.2gb]/[8.5gb]} We're using ES 1.4.2 as a single node cluster, ES\_HEAP is set to 10g, other settings are defaults. From previous posts related to this issue, it is said that field data cache may be a problem. Requesting \*/\_nodes/stats/indices/fielddata \*says: { "cluster\_name": "my\_cluster", "nodes": { "ILUggMfTSvix8Kc0nfNVAw": { "timestamp": 1427188716203, "name": "Danger", "transport\_address": "inet[/192.168.110.91:9300]", "host": "xxx", "ip": [ "inet[/192.168.110.91:9300]", "NONE" ], "indices": { "fielddata": { "memory\_size\_in\_bytes": 64822224, "evictions": 0 } } } } } Running top results in: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java Any ideas what to do? If possible I would rather avoid increasing ES\_HEAP as there isn't that much free memory left on the host. Regards, Abid -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [March 24, 2015, 2:24pm UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848/2 "2015-03-24T14:24:37Z")
what is the new gen setting? how much is the system memory in total? how many cpu cores in the node? what query do you use? jason On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain [hussain@novacom.mygbiz.com](mailto:hussain@novacom.mygbiz.com) wrote: > Hi all, > > we today got (for the first time) warning messages which seem to indicate a > memory problem: > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total > [5m]/[16.7m], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] > [6.9gb]-\>[3.7gb]/[8.5gb]} > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total > [18.4s]/[58s], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] > [6.9gb]-\>[3.7gb]/[8.5gb]} > [2015-03-24 09:08:15,372][WARN][monitor.jvm] [Danger] > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], total > [1.4s]/[16.7m], memory [3.7gb]-\>[5gb]/[9.8gb], all\_pools {[young] > [6.1mb]-\>[2.7mb]/[1.1gb]}{[survivor] [0b]-\>[149.7mb]/[149.7mb]}{[old] > [3.7gb]-\>[4.9gb]/[8.5gb]} > [2015-03-24 09:08:18,192][WARN][monitor.jvm] [Danger] > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], total > [1.4s]/[16.7m], memory [5.8gb]-\>[6.2gb]/[9.8gb], all\_pools {[young] > [845.4mb]-\>[1.2mb]/[1.1gb]}{[survivor] [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > [4.9gb]-\>[6gb]/[8.5gb]} > [2015-03-24 09:08:21,506][WARN][monitor.jvm] [Danger] > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], total > [1.2s]/[16.7m], memory [7gb]-\>[7.3gb]/[9.8gb], all\_pools {[young] > [848.6mb]-\>[2.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > [6gb]-\>[7.2gb]/[8.5gb]} > > We're using ES 1.4.2 as a single node cluster, ES\_HEAP is set to 10g, other > settings are defaults. From previous posts related to this issue, it is said > that field data cache may be a problem. > > Requesting /\_nodes/stats/indices/fielddata says: > { > "cluster\_name": "my\_cluster", > "nodes": { > "ILUggMfTSvix8Kc0nfNVAw": { > "timestamp": 1427188716203, > "name": "Danger", > "transport\_address": "inet[/192.168.110.91:9300]", > "host": "xxx", > "ip": [ > "inet[/192.168.110.91:9300]", > "NONE" > ], > "indices": { > "fielddata": { > "memory\_size\_in\_bytes": 64822224, > "evictions": 0 > } > } > } > } > } > > Running top results in: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java > > Any ideas what to do? If possible I would rather avoid increasing ES\_HEAP as > there isn't that much free memory left on the host. > > Regards, > > Abid > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itz65SQBZu8NOTXgEo6OGp0LdGbTbzvfhp-NPSR%3DaNVXAw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itz65SQBZu8NOTXgEo6OGp0LdGbTbzvfhp-NPSR%3DaNVXAw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Abid\_Hussain](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@Abid\_Hussain](https://discuss.elastic.co/u/Abid_Hussain)\ **Post date:** [March 24, 2015, 3:59pm UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848/3 "2015-03-24T15:59:33Z")
All settings except from ES\_HEAP (set to 10 GB) are defaults, so I actually am not sure about the new gen setting. The host has 80 GB memory in total and 24 CPU cores. All ES indices together sum up to ~32 GB where the biggest indices are of size ~8 GB. We are using queries mostly together with filters and also aggregations. We "solved" the problem with a restart of the cluster. Are there any recommended diagnostics to be performed when this problem occurs next time? Regards, Abid Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee: > what is the new gen setting? how much is the system memory in total? > how many cpu cores in the node? what query do you use? > > jason > > On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain > \<[hus...@novacom.mygbiz.com](mailto:hus...@novacom.mygbiz.com) \\> wrote: > > > Hi all, > > > > we today got (for the first time) warning messages which seem to > > indicate a > > memory problem: > > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total > > [5m]/[16.7m], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] > > [6.9gb]-\>[3.7gb]/[8.5gb]} > > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total > > [18.4s]/[58s], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] > > [6.9gb]-\>[3.7gb]/[8.5gb]} > > [2015-03-24 09:08:15,372][WARN][monitor.jvm] [Danger] > > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], > > total > > [1.4s]/[16.7m], memory [3.7gb]-\>[5gb]/[9.8gb], all\_pools {[young] > > [6.1mb]-\>[2.7mb]/[1.1gb]}{[survivor] [0b]-\>[149.7mb]/[149.7mb]}{[old] > > [3.7gb]-\>[4.9gb]/[8.5gb]} > > [2015-03-24 09:08:18,192][WARN][monitor.jvm] [Danger] > > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], > > total > > [1.4s]/[16.7m], memory [5.8gb]-\>[6.2gb]/[9.8gb], all\_pools {[young] > > [845.4mb]-\>[1.2mb]/[1.1gb]}{[survivor] > > [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > > [4.9gb]-\>[6gb]/[8.5gb]} > > [2015-03-24 09:08:21,506][WARN][monitor.jvm] [Danger] > > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], > > total > > [1.2s]/[16.7m], memory [7gb]-\>[7.3gb]/[9.8gb], all\_pools {[young] > > [848.6mb]-\>[2.1mb]/[1.1gb]}{[survivor] > > [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > > [6gb]-\>[7.2gb]/[8.5gb]} > > > > We're using ES 1.4.2 as a single node cluster, ES\_HEAP is set to 10g, > > other > > settings are defaults. From previous posts related to this issue, it is > > said > > that field data cache may be a problem. > > > > Requesting /\_nodes/stats/indices/fielddata says: > > { > > "cluster\_name": "my\_cluster", > > "nodes": { > > "ILUggMfTSvix8Kc0nfNVAw": { > > "timestamp": 1427188716203, > > "name": "Danger", > > "transport\_address": "inet[/192.168.110.91:9300]", > > "host": "xxx", > > "ip": [ > > "inet[/192.168.110.91:9300]", > > "NONE" > > ], > > "indices": { > > "fielddata": { > > "memory\_size\_in\_bytes": 64822224, > > "evictions": 0 > > } > > } > > } > > } > > } > > > > Running top results in: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java > > > > Any ideas what to do? If possible I would rather avoid increasing > > ES\_HEAP as > > there isn't that much free memory left on the host. > > > > Regards, > > > > Abid > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [March 24, 2015, 11:47pm UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848/4 "2015-03-24T23:47:01Z")
Few filters should be fine but aggregations... hm.... You could use dump stack trace and/or heap dump if it happen again and start to analyze from there. Try as well, > **[Nodes hot threads API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)** hth jason On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain [hussain@novacom.mygbiz.com](mailto:hussain@novacom.mygbiz.com) wrote: > All settings except from ES\_HEAP (set to 10 GB) are defaults, so I actually > am not sure about the new gen setting. > > The host has 80 GB memory in total and 24 CPU cores. All ES indices together > sum up to ~32 GB where the biggest indices are of size ~8 GB. > > We are using queries mostly together with filters and also aggregations. > > We "solved" the problem with a restart of the cluster. Are there any > recommended diagnostics to be performed when this problem occurs next time? > > Regards, > > Abid > > Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee: > > > what is the new gen setting? how much is the system memory in total? > > how many cpu cores in the node? what query do you use? > > > > jason > > > > On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain > > [hus...@novacom.mygbiz.com](mailto:hus...@novacom.mygbiz.com) wrote: > > > > > Hi all, > > > > > > we today got (for the first time) warning messages which seem to > > > indicate a > > > memory problem: > > > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > > > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], total > > > [5m]/[16.7m], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > > > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] > > > [6.9gb]-\>[3.7gb]/[8.5gb]} > > > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > > > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], total > > > [18.4s]/[58s], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > > > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old] > > > [6.9gb]-\>[3.7gb]/[8.5gb]} > > > [2015-03-24 09:08:15,372][WARN][monitor.jvm] [Danger] > > > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], > > > total > > > [1.4s]/[16.7m], memory [3.7gb]-\>[5gb]/[9.8gb], all\_pools {[young] > > > [6.1mb]-\>[2.7mb]/[1.1gb]}{[survivor] [0b]-\>[149.7mb]/[149.7mb]}{[old] > > > [3.7gb]-\>[4.9gb]/[8.5gb]} > > > [2015-03-24 09:08:18,192][WARN][monitor.jvm] [Danger] > > > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], > > > total > > > [1.4s]/[16.7m], memory [5.8gb]-\>[6.2gb]/[9.8gb], all\_pools {[young] > > > [845.4mb]-\>[1.2mb]/[1.1gb]}{[survivor] > > > [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > > > [4.9gb]-\>[6gb]/[8.5gb]} > > > [2015-03-24 09:08:21,506][WARN][monitor.jvm] [Danger] > > > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], > > > total > > > [1.2s]/[16.7m], memory [7gb]-\>[7.3gb]/[9.8gb], all\_pools {[young] > > > [848.6mb]-\>[2.1mb]/[1.1gb]}{[survivor] > > > [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > > > [6gb]-\>[7.2gb]/[8.5gb]} > > > > > > We're using ES 1.4.2 as a single node cluster, ES\_HEAP is set to 10g, > > > other > > > settings are defaults. From previous posts related to this issue, it is > > > said > > > that field data cache may be a problem. > > > > > > Requesting /\_nodes/stats/indices/fielddata says: > > > { > > > "cluster\_name": "my\_cluster", > > > "nodes": { > > > "ILUggMfTSvix8Kc0nfNVAw": { > > > "timestamp": 1427188716203, > > > "name": "Danger", > > > "transport\_address": "inet[/192.168.110.91:9300]", > > > "host": "xxx", > > > "ip": [ > > > "inet[/192.168.110.91:9300]", > > > "NONE" > > > ], > > > "indices": { > > > "fielddata": { > > > "memory\_size\_in\_bytes": 64822224, > > > "evictions": 0 > > > } > > > } > > > } > > > } > > > } > > > > > > Running top results in: > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java > > > > > > Any ideas what to do? If possible I would rather avoid increasing > > > ES\_HEAP as > > > there isn't that much free memory left on the host. > > > > > > Regards, > > > > > > Abid > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itw3Ezh7fxMKaHAJLcK1kPSPhxMOVNNPZX-w0rpPnVmrqg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itw3Ezh7fxMKaHAJLcK1kPSPhxMOVNNPZX-w0rpPnVmrqg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Abid\_Hussain](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@Abid\_Hussain](https://discuss.elastic.co/u/Abid_Hussain)\ **Post date:** [March 26, 2015, 10:38am UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848/5 "2015-03-26T10:38:53Z")
Maybe aggregations are a cause for memory problems. According to the docs, we set the fielddata cache property to indices.fielddata.cache.size: 40% ... hoping this will help to avoid this kind of issues. Regards, Abid Am Mittwoch, 25. März 2015 00:47:09 UTC+1 schrieb Jason Wee: > Few filters should be fine but aggregations... hm.... > > You could use dump stack trace and/or heap dump if it happen again and > start to analyze from there. Try as well, > > [Nodes hot threads API | Elasticsearch Guide [8.11] | Elastic](http://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) > > hth > > jason > > On Tue, Mar 24, 2015 at 11:59 PM, Abid Hussain > \<[hus...@novacom.mygbiz.com](mailto:hus...@novacom.mygbiz.com) \\> wrote: > > > All settings except from ES\_HEAP (set to 10 GB) are defaults, so I > > actually > > am not sure about the new gen setting. > > > > The host has 80 GB memory in total and 24 CPU cores. All ES indices > > together > > sum up to ~32 GB where the biggest indices are of size ~8 GB. > > > > We are using queries mostly together with filters and also aggregations. > > > > We "solved" the problem with a restart of the cluster. Are there any > > recommended diagnostics to be performed when this problem occurs next > > time? > > > > Regards, > > > > Abid > > > > Am Dienstag, 24. März 2015 15:24:43 UTC+1 schrieb Jason Wee: > > > > > what is the new gen setting? how much is the system memory in total? > > > how many cpu cores in the node? what query do you use? > > > > > > jason > > > > > > On Tue, Mar 24, 2015 at 5:26 PM, Abid Hussain > > > [hus...@novacom.mygbiz.com](mailto:hus...@novacom.mygbiz.com) wrote: > > > > > > > Hi all, > > > > > > > > we today got (for the first time) warning messages which seem to > > > > indicate a > > > > memory problem: > > > > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > > > > [gc][young][413224][18109] duration [5m], collections [1]/[5.3m], > > > > total > > > > [5m]/[16.7m], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > > > > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] > > > > [149.7mb]-\>[0b]/[149.7mb]}{[old] > > > > [6.9gb]-\>[3.7gb]/[8.5gb]} > > > > [2015-03-24 09:08:12,960][WARN][monitor.jvm] [Danger] > > > > [gc][old][413224][104] duration [18.4s], collections [1]/[5.3m], > > > > total > > > > [18.4s]/[58s], memory [7.9gb]-\>[3.7gb]/[9.8gb], all\_pools {[young] > > > > [853.9mb]-\>[6.1mb]/[1.1gb]}{[survivor] > > > > [149.7mb]-\>[0b]/[149.7mb]}{[old] > > > > [6.9gb]-\>[3.7gb]/[8.5gb]} > > > > [2015-03-24 09:08:15,372][WARN][monitor.jvm] [Danger] > > > > [gc][young][413225][18110] duration [1.4s], collections [1]/[2.4s], > > > > total > > > > [1.4s]/[16.7m], memory [3.7gb]-\>[5gb]/[9.8gb], all\_pools {[young] > > > > [6.1mb]-\>[2.7mb]/[1.1gb]}{[survivor] [0b]-\>[149.7mb]/[149.7mb]}{[old] > > > > [3.7gb]-\>[4.9gb]/[8.5gb]} > > > > [2015-03-24 09:08:18,192][WARN][monitor.jvm] [Danger] > > > > [gc][young][413227][18111] duration [1.4s], collections [1]/[1.8s], > > > > total > > > > [1.4s]/[16.7m], memory [5.8gb]-\>[6.2gb]/[9.8gb], all\_pools {[young] > > > > [845.4mb]-\>[1.2mb]/[1.1gb]}{[survivor] > > > > [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > > > > [4.9gb]-\>[6gb]/[8.5gb]} > > > > [2015-03-24 09:08:21,506][WARN][monitor.jvm] [Danger] > > > > [gc][young][413229][18112] duration [1.2s], collections [1]/[2.3s], > > > > total > > > > [1.2s]/[16.7m], memory [7gb]-\>[7.3gb]/[9.8gb], all\_pools {[young] > > > > [848.6mb]-\>[2.1mb]/[1.1gb]}{[survivor] > > > > [149.7mb]-\>[149.7mb]/[149.7mb]}{[old] > > > > [6gb]-\>[7.2gb]/[8.5gb]} > > > > > > > > We're using ES 1.4.2 as a single node cluster, ES\_HEAP is set to 10g, > > > > other > > > > settings are defaults. From previous posts related to this issue, it > > > > is > > > > said > > > > that field data cache may be a problem. > > > > > > > > Requesting /\_nodes/stats/indices/fielddata says: > > > > { > > > > "cluster\_name": "my\_cluster", > > > > "nodes": { > > > > "ILUggMfTSvix8Kc0nfNVAw": { > > > > "timestamp": 1427188716203, > > > > "name": "Danger", > > > > "transport\_address": "inet[/192.168.110.91:9300]", > > > > "host": "xxx", > > > > "ip": [ > > > > "inet[/192.168.110.91:9300]", > > > > "NONE" > > > > ], > > > > "indices": { > > > > "fielddata": { > > > > "memory\_size\_in\_bytes": 64822224, > > > > "evictions": 0 > > > > } > > > > } > > > > } > > > > } > > > > } > > > > > > > > Running top results in: > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > > 12735 root 20 0 15.8g 10g 0 S 74 13.2 2485:26 java > > > > > > > > Any ideas what to do? If possible I would rather avoid increasing > > > > ES\_HEAP as > > > > there isn't that much free memory left on the host. > > > > > > > > Regards, > > > > > > > > Abid > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, > > > > send > > > > an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c0116c2-309f-4daf-ad76-b12b866f9066%40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4320ec5a-1dc6-4eec-83c2-d9da618bf957%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fbf022ac-1b94-413c-b7db-0bee0ea6dbe8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fbf022ac-1b94-413c-b7db-0bee0ea6dbe8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:23am UTC](https://discuss.elastic.co/t/encountered-monitor-jvm-warning/22848/6 "2017-07-06T00:23:45Z")