Elasticsearch cause linux kernel crash

Elastic Stack Elasticsearch
1

Hi,guys, thanks for your reading my poor english.
I got a situation that elasticsearch cause my linux kernel crash.However, what information i got from logs(linux messsage log) is Garbled. What's worse i found nothing from my elasticsearch's logs (info level).The crash will happen at least once time each day.

linux message log:
(plenty of that info ,i ask for red hed help, they just say io is too heavey. And the cloud serivices say their software is ok.)
[12655.166750] sd 14:65535:11:0: [sdi] CDB: Write(10) 2a 00 40 87 42 d8 00 01 70 00
[12655.188749] SD100EP: [ERR][epfront_scmd_printk][1096]: scsi_cmnd retrying: serial_number[4192555] retries[1], allowed[5]
[12655.188756] sd 14:65535:11:0: [sdi] CDB: Write(10) 2a 00 40 87 45 78 00 00 50 00
[12655.211419] SD100EP: [ERR][epfront_scmd_printk][1096]: scsi_cmnd retrying: serial_number[4192614] retries[1], allowed[5]
[12655.211424] sd 14:65535:11:0: [sdi] CDB: Write(10) 2a 00 40 87 47 38 00 00 78 00
[12655.224413] SD100EP: [ERR][epfront_scmd_printk][1096]: scsi_cmnd retrying: serial_number[4192637] retries[1], allowed[5]
[12655.224418] sd 14:65535:11:0: [sdi] CDB: Write(10) 2a 00 40 87 47 a8 00 02 70 00

Here is my linux env:
Linux version 3.10.0-514.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11)

12655.400653] SD100EP: [ERR][epfront_io_send][2198]: alloc_iod failed, no memory
[12655.402837] SD100EP: [ERR][epfront_io_send][2198]: alloc_iod failed, no memory

after that ,linux kernel dead.

jvm version:
1.8.0_91-b14
30 g for each instance，each computer run four instance.

hardware:
1556core 256G ram 22t ssd 2*6t sas

logstash conf:
http.port: 9600-9700
pipeline.workers: 80
pipeline.batch.size: 1500
#pipeline.batch.delay: 100
config.reload.automatic: true
config.reload.interval: 3s

elasticsearch conf:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
node.max_local_storage_nodes: 10
thread_pool.get.queue_size: 10000
thread_pool.write.queue_size: 10000
thread_pool.analyze.queue_size: 1000
thread_pool.search.queue_size: 10000
thread_pool.listener.queue_size: 10000
bootstrap.system_call_filter: false
node.attr.disk_type: ssd
node.attr.zone: data-3-90
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: data-2-52,data-3-90,data-3-222,data-3-19,data-2-126,data-2-89,data-2-153，data-2-211,data-2-4,data-2-146,data-3-220,data-2-38
#transport.compress: true

I check my jvm gc suck like is ok. But es takes my memory nerlly 99% of my service(es jvm+buffer). The crash will happen at least once time each day.

2

Which version of Elasticsearch are you running? Is there anything in the Elasticsearch logs?

This seems quite old and not supported according to the support matrix.

3

Thanks for your time. My es version is 7.0.1, and my es log will show as follow(the crash service is data-3-90-1, and this is 3-90 log, what i get sometime) :

[2019-06-26T12:02:42,393][WARN ][o.e.c.a.s.ShardStateAction] [data-3-90-1] node closed while execution action [internal:cluster/shard/failure] for shard entry [shard id [[cloud-edrive-download][2]], allocation id [BR53DpZ1QSy9lFxqzP9-nw], primary term [10], message [failed to perform indices:data/write/bulk[s] on replica [cloud-edrive-download][2], node[mIwqy245SwieG4xwck9k3A], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=BR53DpZ1QSy9lFxqzP9-nw], unassigned_info[[reason=NODE_LEFT], at[2019-06-26T03:39:47.526Z], delayed=false, details[node_left [mIwqy245SwieG4xwck9k3A]], allocation_status[no_attempt]]], failure [NodeDisconnectedException[[data-3-19-1][10.10.3.19:9302][indices:data/write/bulk[s][r]] disconnected]], markAsStale [true]]

[2019-06-26T11:21:32,154][WARN ][o.e.c.NodeConnectionsService] [data-3-90-1] failed to connect to node {data-2-89-2}{f0RQuUDoQKG3AG0KFuZ9jQ}{pohYLyp0RFGZd9PM0YVGCw}{10.10.2.89}{10.10.2.89:9300}{ml.machine_memory=269889724416, ml.max_open_jobs=20, xpack.installed=true, zone=data-2-89, disk_type=sas} (tried [1] times)

what puzzle me is sometime before crash i just got a log few hours ago(gc log，gc is ok，we can see monitoring gc young count is 1/per minute, old is 30 minute. )

I can give you the total log by email .
Really appreciate for your great help,by john.

4

What is the full output of the cluster stats API?

Why have you bumped these up so far?

This may not be directly related to the issue, but as it sems I/O or scsi related I am not able to help there...

5

I/O problem i get from red hat and my cloud service reply is : My service is suffer from system loding. Which cause i/o is terrible(because the disk is not local disk in cloud service, it need few system source to output to my disk. This situation casue only if system loding over 60 by top showing. ).

6

stats api show are as follow(To protect Linux we use cpulimit in each instance just from today. ):
{
"_nodes" : {
"total" : 53,
"successful" : 53,
"failed" : 0
},
"cluster_name" : "elastic",
"cluster_uuid" : "I1z8p5Q-SPe-Y4OFw8AXTQ",
"timestamp" : 1561529621384,
"status" : "red",
"indices" : {
"count" : 1450,
"shards" : {
"total" : 5204,
"primaries" : 4075,
"replication" : 0.27705521472392636,
"index" : {
"shards" : {
"min" : 1,
"max" : 10,
"avg" : 3.5889655172413795
},
"primaries" : {
"min" : 1,
"max" : 5,
"avg" : 2.810344827586207
},
"replication" : {
"min" : 0.0,
"max" : 1.0,
"avg" : 0.27089655172413796
}
}
},
"docs" : {
"count" : 53501026655,
"deleted" : 5322372
},
"store" : {
"size" : "29tb",
"size_in_bytes" : 31969924566435
},
"fielddata" : {
"memory_size" : "226.6kb",
"memory_size_in_bytes" : 232072,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "1.4gb",
"memory_size_in_bytes" : 1576283504,
"total_count" : 138023014,
"hit_count" : 20357130,
"miss_count" : 117665884,
"cache_size" : 29687,
"cache_count" : 75719,
"evictions" : 46032
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 125847,
"memory" : "68gb",
"memory_in_bytes" : 73067214664,
"terms_memory" : "56.2gb",
"terms_memory_in_bytes" : 60417899019,
"stored_fields_memory" : "10.5gb",
"stored_fields_memory_in_bytes" : 11354919656,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "151.9mb",
"norms_memory_in_bytes" : 159356800,
"points_memory" : "1gb",
"points_memory_in_bytes" : 1103197673,
"doc_values_memory" : "30.3mb",
"doc_values_memory_in_bytes" : 31841516,
"index_writer_memory" : "1.4gb",
"index_writer_memory_in_bytes" : 1551804982,
"version_map_memory" : "1.8mb",
"version_map_memory_in_bytes" : 1973242,
"fixed_bit_set" : "23.2mb",
"fixed_bit_set_memory_in_bytes" : 24373568,
"max_unsafe_auto_id_timestamp" : 1561527126190,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 53,
"data" : 48,
"coordinating_only" : 0,
"master" : 5,
"ingest" : 52
},
"versions" : [
"7.0.1"
],
"os" : {
"available_processors" : 2952,
"allocated_processors" : 2952,
"names" : [
{
"name" : "Linux",
"count" : 53
}
],
"pretty_names" : [
{
"pretty_name" : "CentOS Linux 7 (Core)",
"count" : 53
}
],
"mem" : {
"total" : "12.8tb",
"total_in_bytes" : 14168874549248,
"free" : "143gb",
"free_in_bytes" : 153556291584,
"used" : "12.7tb",
"used_in_bytes" : 14015318257664,
"free_percent" : 1,
"used_percent" : 99
}
},
"process" : {
"cpu" : {
"percent" : 111
},
"open_file_descriptors" : {
"min" : 2074,
"max" : 9432,
"avg" : 4674
}
},
"jvm" : {
"max_uptime" : "23.6d",
"max_uptime_in_millis" : 2044555574,
"versions" : [
{
"version" : "1.8.0_91",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "25.91-b14",
"vm_vendor" : "Oracle Corporation",
"bundled_jdk" : true,
"using_bundled_jdk" : false,
"count" : 48
},
{
"version" : "12.0.1",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "12.0.1+12",
"vm_vendor" : "Oracle Corporation",
"bundled_jdk" : true,
"using_bundled_jdk" : true,
"count" : 5
}
],
"mem" : {
"heap_used" : "349.2gb",
"heap_used_in_bytes" : 374988337728,
"heap_max" : "1.4tb",
"heap_max_in_bytes" : 1636081139712
},
"threads" : 17227
},
"fs" : {
"total" : "64.4tb",
"total_in_bytes" : 70832357376000,
"free" : "54.5tb",
"free_in_bytes" : 59953917485056,
"available" : "54.5tb",
"available_in_bytes" : 59953917485056
},
"plugins" : ,
"network_types" : {
"transport_types" : {
"security4" : 53
},
"http_types" : {
"security4" : 53
}
},
"discovery_types" : {
"zen" : 53
}
}
}

7

How many indices and shards are you actively indexing into? How large are your documents? What bulk size are you using?

I can also see that 5 of your nodes are using a bundled newer JVM. All nodes should run the same version so I would recommend switching to the more up to date and supported one that is bundled with the distribution.

based on the stats it looks like you have reasonably large documents and that you may be updating them as well as indexing. Is this correct? If so, can you describe the load and use case in more detail?

8

Your are right .The lagest index is nearly 1 tb(contain ont Replica) with 5 parimary shard . In most case we control the index with 5 shard , and the size limit 50g(each shard without Replica.). We are trying to using rollover to control the shard size.

9

Thanks for your help. Firstly , i will fix the java version.
We input our log by logstash to es , which daily insert nerly 1 tb( original data)（in the coming day ,we will input 3 tb per/day），here is our template setting:
"settings" : {
"index" : {
"lifecycle" : {
"name" : "cloud-edrive-download",
"rollover_alias" : "cloud-edrive-download"
},
"routing" : {
"allocation" : {
"require" : {
"disk_type" : "ssd"
},
"total_shards_per_node" : "2"
}
},
"refresh_interval" : "10s",
"number_of_shards" : "5",
"number_of_replicas" : "1"

What's worse we just have 12 * 2t ssd. So we just can using 6 instance as the hot node(each node with 2 * 2 tb ssd).

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.