Cluster setup and monitoring with xpack in kibana - ELK 5.0

[marimuthu (Mari Muthu M)]

Hi,

I'm learning the ELK stack to setup a cluster and monitor the same using x-pack plugin. When i dedicately create nodes for different purpose it fails. I'm running all these in one machine with Windows 7 and 8GB of RAM.

With the available the documents in the elastic.co site i made the below configurations for nodes in a cluster.
I've installed x-pack plugin in all the elastic nodes and kibana. The only other change i made to all nodes is jvm.options - [-Xms1g -Xmx1g]

Please help me in resolving the issue. your help is highly appreciated.

Common configurations in all nodes

cluster.name: dev-cluster
network.host: 172.24.5.33
xpack.security.enabled: false
xpack.monitoring.enabled: true

Master Node

http.port: 9200
node.master: true
node.ingest: false

Data node:

node.master: false
node.data: true
node.ingest: false
http.port: 9201

Client Node:

node.master: false
node.data: false
node.ingest: false
http.port: 9205

I've one another node (running in 9202 port) that has only the common config mentioned above. Kibana points to the client node. Because of the below errors its status is red.

Error 1: In Master Node

   ... 4 more

2017-01-31T19:23:51,324][ERROR][o.e.x.m.AgentService ] [node-master] exception when exporting documents
rg.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:148) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.close(ExportBulk.java:77) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:194) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.0.0.jar:5.0.0]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
aused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulk [default_local]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.doFlush(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk.flush(ExportBulk.java:62) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.doFlush(ExportBulk.java:145) ~[?:?]
... 4 more
aused by: java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.
at org.elasticsearch.action.ingest.IngestProxyActionFilter.randomIngestNode(IngestProxyActionFilter.java:104) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.ingest.IngestProxyActionFilter.forwardIngestRequest(IngestProxyActionFilter.java:86) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.ingest.IngestProxyActionFilter.apply(IngestProxyActionFilter.java:73) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:180) ~[elasticsearch-5.0.0.jar:5.0.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:153) ~[elasticsearch-5.0.0.jar:5.0.0]

Error 2: In client node

[2017-01-31T19:25:31,071][INFO ][o.e.x.m.e.Exporters ] [node-client] skipping exporter [default_local] as it isn't ready yet
[2017-01-31T19:25:31,072][ERROR][o.e.x.m.AgentService ] [node-client] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: exporters are either not ready or faulty
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:188) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.0.0.jar:5.0.0]

Error 3: In data node

    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]

[2017-01-31T19:39:01,607][INFO ][o.e.x.m.e.Exporters ] [node-data] skipping exporter [default_local] as it isn't ready yet
[2017-01-31T19:39:01,608][ERROR][o.e.x.m.AgentService ] [node-data] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: exporters are either not ready or faulty
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:188) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.0.0.jar:5.0.0]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]

[cbuescher (Christoph)]

Hi,

Running four distinct node on a single laptop with only that amount if memory is a bit of a stretch. If you are just checking out Elasticsearch you should probably start with one or two nodes only and let them all be able to take the master/data/ingest roles. If you want to practice a more realistic cluster setup I'd suggest to move to more than one machine.

That said, your master reports that it doesn't find any nodes that can act as "ingest" nodes:

java.lang.IllegalStateException: There are no ingest nodes in this cluster, unable to forward request to an ingest node.

Can you try setting the node.ingest: true for one of the nodes (e.g. master)?

[marimuthu (Mari Muthu M)]

Hello,

Thanks for your quick response. I tried with setting the node.ingest: true for the one running at 9202

Logs
C:\Data\software\elk5.0\elasticSearch-3\bin>elasticsearch
[2017-01-31T20:07:26,887][INFO ][o.e.n.Node ] [node-ingest] initializing ...
[2017-01-31T20:07:27,262][INFO ][o.e.e.NodeEnvironment ] [node-ingest] using [1] data paths, mounts [[Windows (C:)]], net usable_space [57.6gb], net total_space [147gb], spi
ns? [unknown], types [NTFS]
[2017-01-31T20:07:27,276][INFO ][o.e.e.NodeEnvironment ] [node-ingest] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-01-31T20:07:28,152][INFO ][o.e.n.Node ] [node-ingest] version[5.0.0], pid[19540], build[253032b/2016-10-26T04:37:51.531Z], OS[Windows 7/6.1/amd64], JVM[Orac
le Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_25/25.25-b02]
[2017-01-31T20:07:32,349][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [aggs-matrix-stats]
[2017-01-31T20:07:32,357][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [ingest-common]
[2017-01-31T20:07:32,378][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [lang-expression]
[2017-01-31T20:07:32,405][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [lang-groovy]
[2017-01-31T20:07:32,411][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [lang-mustache]
[2017-01-31T20:07:32,448][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [lang-painless]
[2017-01-31T20:07:32,473][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [percolator]
[2017-01-31T20:07:32,481][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [reindex]
[2017-01-31T20:07:32,509][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [transport-netty3]
[2017-01-31T20:07:32,510][INFO ][o.e.p.PluginsService ] [node-ingest] loaded module [transport-netty4]
[2017-01-31T20:07:32,511][INFO ][o.e.p.PluginsService ] [node-ingest] loaded plugin [x-pack]
[2017-01-31T20:07:42,189][INFO ][o.e.n.Node ] [node-ingest] initialized
[2017-01-31T20:07:42,190][INFO ][o.e.n.Node ] [node-ingest] starting ...
[2017-01-31T20:07:42,837][INFO ][o.e.t.TransportService ] [node-ingest] publish_address {172.24.5.33:9302}, bound_addresses {172.24.5.33:9302}
[2017-01-31T20:07:42,858][INFO ][o.e.b.BootstrapCheck ] [node-ingest] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-31T20:07:52,392][INFO ][o.e.x.m.e.Exporters ] [node-ingest] skipping exporter [default_local] as it isn't ready yet
[2017-01-31T20:07:52,393][ERROR][o.e.x.m.AgentService ] [node-ingest] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: exporters are either not ready or faulty
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:188) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.0.0.jar:5.0.0]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
[2017-01-31T20:08:02,433][INFO ][o.e.x.m.e.Exporters ] [node-ingest] skipping exporter [default_local] as it isn't ready yet
[2017-01-31T20:08:02,434][ERROR][o.e.x.m.AgentService ] [node-ingest] exception when exporting documents
org.elasticsearch.xpack.monitoring.exporter.ExportException: exporters are either not ready or faulty
at org.elasticsearch.xpack.monitoring.exporter.Exporters.export(Exporters.java:188) ~[x-pack-5.0.0.jar:5.0.0]
at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:208) [x-pack-5.0.0.jar:5.0.0]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
[2017-01-31T20:08:12,442][INFO ][o.e.x.m.e.Exporters ] [node-ingest] skipping exporter [default_local] as it isn't ready yet

Still my master node shows the same error as in the main issue.

[cbuescher (Christoph)]

I would enable ingest on all nodes and restart to be honest, unless there is something particular you want to try out by disabling it.