Currently i have a cluster deployed on EC2 from AWS. There are 3 nodes.
Here are the config from each node:
node1
cluster.name: woombat
cluster.initial_master_nodes: ["node1"]
node.name: node1
node.master: true
network.host: 0.0.0.0
transport.tcp.port: 9300
discovery.seed_hosts: [172.31.15.29,172.31.27.52,172.31.16.248]
node2
cluster.name: woombat
cluster.initial_master_nodes: ["node1"]
node.name: node1
node.master: true
network.host: 0.0.0.0
transport.tcp.port: 9300
discovery.seed_hosts: [172.31.15.29,172.31.27.52,172.31.16.248]
node3
cluster.name: woombat
cluster.initial_master_nodes: ["node1"]
node.name: node3
node.master: true
node.voting_only: true
network.host: 0.0.0.0
transport.tcp.port: 9300
discovery.seed_hosts: [172.31.15.29,172.31.27.52,172.31.16.248]
I enabled the monitoring from kibana and after down the node3 i'm unable to restart elasticsearch in that node. I'm getting the following warning:
[2020-02-26T01:05:40,276][INFO ][o.e.h.AbstractHttpServerTransport] [node3] publish_address {172.31.16.248:9200}, bound_addresses {[::]:9200}
[2020-02-26T01:05:40,283][INFO ][o.e.n.Node ] [node3] started
[2020-02-26T01:10:12,850][INFO ][o.e.c.s.ClusterSettings ] [node3] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-02-26T01:10:23,004][WARN ][o.e.x.m.e.l.LocalExporter] [node3] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[index [.monitoring-es-7-2020.02.26] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.3.1.jar:7.3.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-7.3.1.jar:7.3.1]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-7.3.1.jar:7.3.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:473) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:468) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:74) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:822) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:663) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:165) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:443) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:556) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:261) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:610) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:364) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.1.jar:7.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.1.jar:7.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Any idea how to fix it?