Elasticsearch-5.2.0 java.lang.OutOfMemoryError: Java heap space

Elastic Stack Elasticsearch
1

Elasticsearch version:5.2.0

Plugins installed: []

JVM version:1.8.0_121

OS version:redhat

config/jvm.options

-Xms20g
-Xmx20g

-XX:+UseCondCardMark
-XX:CMSWaitDuration=250

-XX:+UseG1GC
-XX:MaxGCPauseMillis=200

=============================================================================

I need to configure how to solve this problem?
and how to optimize index?

[2017-02-25T22:24:23,840][INFO ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][young][2829][54] duration [1.6s], collections [2]/[2.4s], total [1.6s]/[8.1s], memory [10.6gb]->[6.2gb]/[20gb], all_pools {[young] [8.8gb]->[8mb]/[0b]}{[survivor] [40mb]->[72mb]/[0b]}{[old] [1.7gb]->[6.2gb]/[20gb]}
[2017-02-25T22:24:23,841][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2829] overhead, spent [1.6s] collecting in the last [2.4s]
[2017-02-25T22:24:24,841][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2830] overhead, spent [653ms] collecting in the last [1s]
[2017-02-25T22:24:25,869][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2831] overhead, spent [591ms] collecting in the last [1s]
[2017-02-25T22:24:26,908][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2832] overhead, spent [710ms] collecting in the last [1s]
[2017-02-25T22:24:27,957][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2833] overhead, spent [703ms] collecting in the last [1s]
[2017-02-25T22:24:29,070][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2834] overhead, spent [733ms] collecting in the last [1.1s]
[2017-02-25T22:24:30,071][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2835] overhead, spent [747ms] collecting in the last [1s]
[2017-02-25T22:24:39,251][INFO ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][old][2836][1] duration [8.1s], collections [1]/[9.1s], total [8.1s]/[8.1s], memory [17.9gb]->[18.1gb]/[20gb], all_pools {[young] [304mb]->[8mb]/[0b]}{[survivor] [128mb]->[0b]/[0b]}{[old] [17.5gb]->[18.1gb]/[20gb]}
[2017-02-25T22:24:39,251][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2836] overhead, spent [8.8s] collecting in the last [9.1s]
[2017-02-25T22:24:42,208][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2837] overhead, spent [2.7s] collecting in the last [2.9s]
[2017-02-25T22:24:44,980][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2838] overhead, spent [2.6s] collecting in the last [2.7s]
[2017-02-25T22:24:47,093][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2839] overhead, spent [2s] collecting in the last [2.1s]
[2017-02-25T22:24:48,403][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2840] overhead, spent [1.3s] collecting in the last [1.3s]
[2017-02-25T22:26:55,561][WARN ][o.e.m.j.JvmGcMonitorService] [node-3] [gc][2841] overhead, spent [1.2m] collecting in the last [2.1m]
[2017-02-25T22:26:55,584][WARN ][o.e.d.z.PublishClusterStateAction] [node-3] received a cluster state from a different master than the current one, rejecting (received {node-4}{5BRY8-u3Tj-K274V8qT0wg}{3OGNrcBvS424iNK6BQwLgw}{125.39.176.88}{125.39.176.88:9300}, current {node-2}{S77O9WbzR_Sv0R2tJXgUfQ}{doWv8sGUSOGPuvUv-uEvZA}{125.39.176.86}{125.39.176.86:9300})
[2017-02-25T22:26:55,610][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][search][T#15]], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,609][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][generic][T#3]], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,609][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][refresh][T#3]], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,559][WARN ][i.n.u.c.SingleThreadEventExecutor] Unexpected exception from an event executor:
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,609][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][generic][T#5]], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,609][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][generic][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,564][WARN ][i.n.c.n.NioEventLoop ] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,560][WARN ][i.n.c.n.NioEventLoop ] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,561][WARN ][i.n.c.n.NioEventLoop ] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,564][WARN ][i.n.u.c.SingleThreadEventExecutor] Unexpected exception from an event executor:
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,557][WARN ][i.n.c.n.NioEventLoop ] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,564][WARN ][i.n.c.n.NioEventLoop ] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,609][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-3] fatal error in thread [elasticsearch[node-3][generic][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
[2017-02-25T22:26:55,557][WARN ][i.n.c.n.NioEventLoop ] Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space

2

How many nodes, how many indices/shards/GB of data.

We do not support G1GC.

3

{
"_nodes": {
"total": 4,
"successful": 4,
"failed": 0
},
"cluster_name": "xxxx",
"nodes": {
"S77O9WbzR_Sv0R2tJXgUfQ": {
"timestamp": 1488103776255,
"name": "node-2",
"transport_address": "xxxx:xxx",
"indices": {
"docs": {
"count": 127359010,
"deleted": 0
},
"store": {
"size_in_bytes": 44671418610,
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 41720888,
"index_time_in_millis": 13529348,
"index_current": 2,
"index_failed": 0,
"delete_total": 0,
"delete_time_in_millis": 0,
"delete_current": 0,
"noop_update_total": 0,
"is_throttled": false,
"throttle_time_in_millis": 0
},
"get": {
"total": 0,
"time_in_millis": 0,
"exists_total": 0,
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 0,
"query_total": 13173,
"query_time_in_millis": 1221,
"query_current": 0,
"fetch_total": 13173,
"fetch_time_in_millis": 839,
"fetch_current": 0,
"scroll_total": 0,
"scroll_time_in_millis": 0,
"scroll_current": 0,
"suggest_total": 0,
"suggest_time_in_millis": 0,
"suggest_current": 0
},
"merges": {
"current": 0,
"current_docs": 0,
"current_size_in_bytes": 0,
"total": 11736,
"total_time_in_millis": 12061877,
"total_docs": 192034822,
"total_size_in_bytes": 80859329628,
"total_stopped_time_in_millis": 0,
"total_throttled_time_in_millis": 3719974,
"total_auto_throttle_in_bytes": 199076943
},
"refresh": {
"total": 107198,
"total_time_in_millis": 3128729,
"listeners": 0
},
"flush": {
"total": 41,
"total_time_in_millis": 14658
},
"warmer": {
"current": 0,
"total": 107251,
"total_time_in_millis": 33921
},
"query_cache": {
"memory_size_in_bytes": 0,
"total_count": 0,
"hit_count": 0,
"miss_count": 0,
"cache_size": 0,
"cache_count": 0,
"evictions": 0
},
"segments": {
"count": 228,
"memory_in_bytes": 137908043,
"terms_memory_in_bytes": 123399455,
"stored_fields_memory_in_bytes": 12278768,
"term_vectors_memory_in_bytes": 0,
"norms_memory_in_bytes": 188416,
"points_memory_in_bytes": 1481044,
"doc_values_memory_in_bytes": 560360,
"index_writer_memory_in_bytes": 6080360,
"version_map_memory_in_bytes": 104650,
"fixed_bit_set_memory_in_bytes": 0,
"max_unsafe_auto_id_timestamp": 1488067301360,
"file_sizes": {}
},
"translog": {
"operations": 2937300,
"size_in_bytes": 1573732542
},
"request_cache": {
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
},
"recovery": {
"current_as_source": 0,
"current_as_target": 0,
"throttle_time_in_millis": 596351
}
},
"os": {
"timestamp": 1488103776378,
"cpu": {
"percent": 3,
"load_average": {
"1m": 0.86,
"5m": 0.95,
"15m": 0.92
}
},
"mem": {
"total_in_bytes": 33651437568,
"free_in_bytes": 202268672,
"used_in_bytes": 33449168896,
"free_percent": 1,
"used_percent": 99
},
"swap": {
"total_in_bytes": 67108855808,
"free_in_bytes": 52870606848,
"used_in_bytes": 14238248960
}
},
"process": {
"timestamp": 1488103776379,
"open_file_descriptors": 635,
"max_file_descriptors": 65536,
"cpu": {
"percent": 3,
"total_in_millis": 27088000
},
"mem": {
"total_virtual_in_bytes": 79897645056
}
},
"jvm": {
"timestamp": 1488103776379,
"uptime_in_millis": 67822496,
"mem": {
"heap_used_in_bytes": 5326138248,
"heap_used_percent": 24,
"heap_committed_in_bytes": 21474836480,
"heap_max_in_bytes": 21474836480,
"non_heap_used_in_bytes": 106772416,
"non_heap_committed_in_bytes": 113504256,
"pools": {
"young": {
"used_in_bytes": 2357198848,
"max_in_bytes": 0,
"peak_used_in_bytes": 12876513280,
"peak_max_in_bytes": 0
},
"survivor": {
"used_in_bytes": 33554432,
"max_in_bytes": 0,
"peak_used_in_bytes": 318767104,
"peak_max_in_bytes": 0
},
"old": {
"used_in_bytes": 2935384968,
"max_in_bytes": 21474836480,
"peak_used_in_bytes": 2935384968,
"peak_max_in_bytes": 21474836480
}
}
},
"threads": {
"count": 171,
"peak_count": 175
},
"gc": {
"collectors": {
"young": {
"collection_count": 375,
"collection_time_in_millis": 12972
},
"old": {
"collection_count": 0,
"collection_time_in_millis": 0
}
}
},

4

"buffer_pools": {
"direct": {
"count": 157,
"used_in_bytes": 837149048,
"total_capacity_in_bytes": 837149047
},
"mapped": {
"count": 344,
"used_in_bytes": 41997719220,
"total_capacity_in_bytes": 41997719220
}
},
"classes": {
"current_loaded_count": 10785,
"total_loaded_count": 10785,
"total_unloaded_count": 0
}
},
"thread_pool": {
"bulk": {
"threads": 24,
"queue": 0,
"active": 2,
"rejected": 0,
"largest": 24,
"completed": 124761
},
"fetch_shard_started": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"fetch_shard_store": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 12,
"completed": 19
},
"flush": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 43
},
"force_merge": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"generic": {
"threads": 6,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 6,
"completed": 124107
},
"get": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"index": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"listener": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"management": {
"threads": 3,
"queue": 0,
"active": 1,
"rejected": 0,
"largest": 3,
"completed": 31539
},
"refresh": {
"threads": 4,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 4,
"completed": 207089
},
"search": {
"threads": 37,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 37,
"completed": 13173
},
"snapshot": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"warmer": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 107250
}
},
"fs": {
"timestamp": 1488103776380,
"total": {
"total_in_bytes": 1114021974016,
"free_in_bytes": 972529090560,
"available_in_bytes": 915940016128,
"spins": "true"
},
"data": [
{
"path": "/home/elk/server/soft/elasticsearch-5.2.0/data/nodes/0",
"mount": "/ (/dev/sda3)",
"type": "ext4",
"total_in_bytes": 1114021974016,
"free_in_bytes": 972529090560,
"available_in_bytes": 915940016128,
"spins": "true"
}
],
"io_stats": {
"devices": [
{
"device_name": "sda3",
"operations": 2569456,
"read_operations": 217856,
"write_operations": 2351600,
"read_kilobytes": 13513832,
"write_kilobytes": 132245960
}
],
"total": {
"operations": 2569456,
"read_operations": 217856,
"write_operations": 2351600,
"read_kilobytes": 13513832,
"write_kilobytes": 132245960
}
}
},
"transport": {
"server_open": 39,
"rx_count": 414484,
"rx_size_in_bytes": 52822731962,
"tx_count": 414481,
"tx_size_in_bytes": 18530101416
},
"breakers": {
"request": {
"limit_size_in_bytes": 12884901888,
"limit_size": "12gb",
"estimated_size_in_bytes": 0,
"estimated_size": "0b",
"overhead": 1,
"tripped": 0
},
"fielddata": {
"limit_size_in_bytes": 12884901888,
"limit_size": "12gb",
"estimated_size_in_bytes": 0,
"estimated_size": "0b",
"overhead": 1.03,
"tripped": 0
},
"in_flight_requests": {
"limit_size_in_bytes": 21474836480,
"limit_size": "20gb",
"estimated_size_in_bytes": 778574,
"estimated_size": "760.3kb",
"overhead": 1,
"tripped": 0
},
"parent": {
"limit_size_in_bytes": 15032385536,
"limit_size": "14gb",
"estimated_size_in_bytes": 778574,
"estimated_size": "760.3kb",
"overhead": 1,
"tripped": 0
}
},
},

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.