Elastic node hangs (i think)

Elastic Stack Elasticsearch
1

I just installed elastic search elasticsearch-6.1.3 on RHEL (Red Hat Enterprise Linux Server release 7.4 (Maipo)). I have 2 servers where log stash is installed. I m parsing HTTP server logs successfully. Issue is after some time ES just doesn't do anything. Logstash is still running . I have both logstash running in debug mode and i see log getting updated. ES is up and running and i can do curl

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2018.02.06 NQbQo-QkTdqQhwdsIQpUMw 5 1 12473 0 2.2mb 2.2mb
yellow open logstash-2018.02.04 9W2VRUoZQcW_Gp_pP7V1rQ 5 1 19291 0 2.3mb 2.3mb
yellow open logstash-2018.02.05 chEwnCioTJKZnbq14LrMsw 5 1 8408 0 1.4mb 1.4mb
yellow open logstash-2018.02.03 oExmnZhzQyW9UgqZngD9VQ 5 1 51866 0 6mb 6mb
yellow open .kibana ccvDflniSbSQNFaw3Q8igQ 1 1 2 1 11.3kb 11.3kb
yellow open logstash-2018.02.02 VhhO4YwzSxWhPlfxjFm88w 5 1 11508 0 1.4mb 1.4mb

but line 1 has 12473 documents and it has been like that for 3 hours. Moment i restart ES it changes. I just restarted ES and first line has # docs changed to 18092.

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2018.02.06 NQbQo-QkTdqQhwdsIQpUMw 5 1 18092 0 2.8mb 2.8mb
yellow open logstash-2018.02.04 9W2VRUoZQcW_Gp_pP7V1rQ 5 1 19291 0 2.3mb 2.3mb
yellow open logstash-2018.02.05 chEwnCioTJKZnbq14LrMsw 5 1 8408 0 1.4mb 1.4mb
yellow open logstash-2018.02.03 oExmnZhzQyW9UgqZngD9VQ 5 1 51866 0 6mb 6mb
yellow open .kibana ccvDflniSbSQNFaw3Q8igQ 1 1 2 1 11.3kb 11.3kb
yellow open logstash-2018.02.02 VhhO4YwzSxWhPlfxjFm88w 5 1 11508 0 1.4mb 1.4mb

There is no error in ES logs. I just dont know where to look and what to look :frowning:

  • I m pretty new to to ES *

Raj

2

Ok, so what do your various logs show?

3

so for today there are 3 logs.

-rw-rw-r-- 1 kibana kibana 1845 Feb 6 11:21 msn_cluster-2018-02-05-1.log.gz
-rw-rw-r-- 1 kibana kibana 44532 Feb 6 18:16 msn_cluster_deprecation.log
-rw-rw-r-- 1 kibana kibana 553304 Feb 6 18:16 msn_cluster.log

I enabled debug at 17:34 . then at 18:16 i see this error.

[2018-02-06T18:16:08,219][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [msn_node] failed to put mappings on indices [[[logstash-2018.02.07/qgViCjT1RSmN2ZHxyP6-Iw]]], type [doc]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$null$0(MasterService.java:122) ~[elasticsearch-6.1.3.jar:6.1.3]
at java.util.ArrayList.forEach(ArrayList.java:1255) ~[?:1.8.0_151]
at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:121) ~[elasticsearch-6.1.3.jar:6.1.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.3.jar:6.1.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2018-02-06T18:16:08,248][DEBUG][o.e.c.s.MasterService ] [msn_node] processing [put-mapping[doc]]: execute

then last 2 lines.

[2018-02-06T18:16:08,976][DEBUG][o.e.i.e.Engine ] [msn_node] [logstash-2018.02.06][3] successfully sync committed. sync id [fYvJ7fo7S7u01QTnjBWSnw].
[2018-02-06T18:16:08,984][DEBUG][o.e.i.e.Engine ] [msn_node] [logstash-2018.02.06][4] successfully sync committed. sync id [AQtAxiBWTz-dIZ3K0g8g1A].

4

since last night it has been running and no new data no errors in ES logs.

document count is same !!

i just dont know what the issue is :frowning:

5

out of frustration installed 6.2 . starting new thread.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.