Logstash cannot retrieve license information from server

NadyLeez123 · July 20, 2022, 2:03pm

@sscarduzio
There is more than a week that we are having issue with our ELK 7.15.1. Before that, everything were working well since months.
Here is the problem: when logstash is stopped, the elasticsearch cluster is healthy, kibana is accessible. But once logstash is started, here is its outpout:

[2022-07-20T09:03:41,867][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>50, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>100, "pipeline.sources"=>["/etc/logstash/pipeline.global.conf"], :thread=>"#<Thread:0x71385f3a@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[2022-07-20T09:03:43,539][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.67}
[2022-07-20T09:03:43,669][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2022-07-20T09:03:43,776][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,783][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-07-20T09:03:43,790][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-07-20T09:03:43,797][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,799][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,825][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,826][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,836][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:43,837][INFO ][filewatch.observingread  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2022-07-20T09:03:44,088][INFO ][org.logstash.beats.Server][main] Starting server on port: 5044
[2022-07-20T09:04:05,392][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://user:xxxxxx@master1:9200/_xpack'"}
[2022-07-20T09:04:35,390][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://user:xxxxxx@master1:9200/_xpack'"}
[2022-07-20T09:04:44,198][WARN ][logstash.outputs.elasticsearch][main] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://user:xxxxxx@master2:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://user:xxxxxx@master2:9200/, :error_message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master2.gestion.mrqgest:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2022-07-20T09:04:44,201][ERROR][logstash.outputs.elasticsearch][main] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master2:9200/][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2022-07-20T09:04:44,201][WARN ][logstash.outputs.elasticsearch][main] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://user:xxxxxx@master1:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://user:xxxxxx@master1:9200/, :error_message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master1:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2022-07-20T09:04:44,206][ERROR][logstash.outputs.elasticsearch][main] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://user:xxxxxx@master1:9200/][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>2}
[2022-07-20T09:04:45,686][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://user:xxxxxx@master1:9200/"}

Then, here is the elasticsearch output when logstash is started:

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:56) ~[?:?]
        at org.elasticsearch.xpack.monitoring.action.TransportMonitoringBulkAction.doExecute(TransportMonitoringBulkAction.java:36) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) [elasticsearch-7.15.1.jar:7.15.1]
        at tech.beshu.ror.es.handler.RegularRequestHandler.proceed(RegularRequestHandler.scala:224) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.onAllow(RegularRequestHandler.scala:99) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$commitResult$1(RegularRequestHandler.scala:70) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [scala-library-2.12.10.jar:?]
        at scala.util.Try$.apply(Try.scala:213) [scala-library-2.12.10.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.commitResult(RegularRequestHandler.scala:68) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$2(RegularRequestHandler.scala:60) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$2$adapted(RegularRequestHandler.scala:59) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.utils.ScalaOps$AutoCloseableOps$.bracket$extension(ScalaOps.scala:168) [core-1.36.0.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$1(RegularRequestHandler.scala:59) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.handler.RegularRequestHandler.$anonfun$handle$1$adapted(RegularRequestHandler.scala:58) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at monix.eval.Task$Map.apply(Task.scala:4514) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task$Map.apply(Task.scala:4510) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.internal.TaskRunLoop$.startLight(TaskRunLoop.scala:331) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task.runAsyncOptF(Task.scala:811) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task.runAsyncOpt(Task.scala:709) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at monix.eval.Task.runAsync(Task.scala:659) [monix-eval_2.12-3.0.0.jar:3.0.0]
        at tech.beshu.ror.es.IndexLevelActionFilter.handleRequest(IndexLevelActionFilter.scala:136) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.proceedByRorEngine(IndexLevelActionFilter.scala:122) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.$anonfun$apply$1(IndexLevelActionFilter.scala:107) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [scala-library-2.12.10.jar:?]
        at tech.beshu.ror.utils.AccessControllerHelper$$anon$1.run(AccessControllerHelper.scala:25) [core-1.36.0.jar:?]
        at java.security.AccessController.doPrivileged(AccessController.java:318) [?:?]
        at tech.beshu.ror.utils.AccessControllerHelper$.doPrivileged(AccessControllerHelper.scala:24) [core-1.36.0.jar:?]
        at tech.beshu.ror.es.IndexLevelActionFilter.apply(IndexLevelActionFilter.scala:93) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:90) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:70) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.xpack.monitoring.rest.action.RestMonitoringBulkAction.lambda$doPrepareRequest$0(RestMonitoringBulkAction.java:102) [x-pack-monitoring-7.15.1.jar:7.15.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:106) [elasticsearch-7.15.1.jar:7.15.1]
        at tech.beshu.ror.es.ReadonlyRestPlugin.$anonfun$getRestHandlerWrapper$2(ReadonlyRestPlugin.scala:238) [readonlyrest-1.36.0_es7.15.1.jar:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:275) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:343) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:196) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:348) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:413) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:330) [elasticsearch-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.15.1.jar:7.15.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:17) [transport-netty4-client-7.15.1.jar:7.15.1]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:47) [transport-netty4-client-7.15.1.jar:7.15.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.69.Final.jar:4.1.69.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.69.Final.jar:4.1.69.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]

Any idea about this issue?

Note: in kibana.yml, we have :

xpack.monitoring.enabled: true

and in logstash pipeline:

ssl_certificate_verification => false

Note that everything were working well since months

Thanks

leandrojmp · July 20, 2022, 2:37pm

Is your Elasticsearch cluster running? This log says that there is no master available.

What is the result when you make a curl to https://master1:9200 ?

NadyLeez123 · July 20, 2022, 6:01pm

Yes the elasticsearch cluster is running. The problem occurs when I start logstash. If I stop Logstash, the org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/2/no master] error disappears in elasticsearch logs.

I just stopped logstash, and everything is fine:

curl to https://master1:9200:

{
  "cluster_name" : "elasticsearch_xxxxx",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 8,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 4237,
  "active_shards" : 4850,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Then, if I start Logstash, here is what curl to https://master1:9200 gives:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

So the problem occurs when logstash is running

system · August 17, 2022, 6:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.