Entering settings/reload field list hangs elastic search


(Damian Melniczuk) #1

Hi,
I have working ELK stack which is handling data nicely. I added new type in Logstash and if I want use fields from it in Kibana I have to refresh mappings because "No cached mapping for this field". When I'm entering settings and picking "logstash-*" and clicking "Reload field list" Kibana says that there is no living connection to Elasticseach. I also have message from Logstash that he can't connect Elasticsearch. When I restart Elasticsearch Kibana can run again but fields are still no mapped, so I'm at beginning. Elasticsearch sometimes crash even just after entering "Settings".

Couple of last lines from elasticsearch.log:

[2015-12-07 13:57:33,840][INFO ][monitor.jvm ] [Ripfire] [gc][old][671][44] duration [6.6s], collections [1]/[7s], total [6.6s]/[1.4m], memory [1.8gb]->[1.3gb]/[1.9gb], all_pools {[young] [489.3mb]->[763.5kb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [1.3gb]->[1.3gb]/[1.3gb]}
[2015-12-07 14:01:18,181][INFO ][monitor.jvm ] [Ripfire] [gc][old][885][70] duration [7.3s], collections [1]/[7.7s], total [7.3s]/[1.7m], memory [1.6gb]->[1.3gb]/[1.9gb], all_pools {[young] [294.1mb]->[9.3mb]/[532.5mb]}{[survivor] [33mb]->[0b]/[66.5mb]}{[old] [1.3gb]->[1.3gb]/[1.3gb]}
[2015-12-07 14:01:51,463][INFO ][monitor.jvm ] [Ripfire] [gc][old][908][75] duration [10s], collections [2]/[10.6s], total [10s]/[1.9m], memory [1.7gb]->[1.4gb]/[1.9gb], all_pools {[young] [453mb]->[57mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old] [1.3gb]->[1.3gb]/[1.3gb]}
[2015-12-07 14:02:01,200][INFO ][monitor.jvm ] [Ripfire] [gc][old][910][76] duration [8.4s], collections [1]/[8.6s], total [8.4s]/[2.1m], memory [1.8gb]->[1.9gb]/[1.9gb], all_pools {[young] [532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [3.8mb]->[49.3mb]/[66.5mb]}{[old] [1.3gb]->[1.3gb]/[1.3gb]}

Versions of used software:

  • Elasticsearch 1.7.2
  • Logstash 1.5.4
  • Kibana 4.1.2

(Damian Melniczuk) #2

I just found in logs:

Caused by: java.lang.OutOfMemoryError: Java heap space

so I updated ES_HEAP_SIZE=4g from to 2g and it works correctly now :slight_smile:


(Mark Walkom) #3

Yeah, your GC is massive so you needed more heap :slight_smile:


(system) #4