Waiting for admin cluster to become reachable, after OS upgrade from rhek 7.6 to rhel 7.7

[logger (Malte Leek)]

Hi there,

I had to upgrade the OS and for this I deleted the ECE completely like mentioned in the docs. It worked and I reinstalled and rejoined the node. In the UI I can see the runner and configure the roles. But it seems that it has an issue with the admin cluster and the Proxy.

 [2019-11-11 11:17:06,115][WARN ][spray.can.client.HttpHostConnectionSlot] Connection attempt to containerhost:9244 failed in response to POST request to /allocator-metricbeat-*/_search with no retries left, dispatching error... {}
[2019-11-11 11:17:06,549][WARN ][spray.can.client.HttpHostConnectionSlot] Connection attempt to containerhost:9244 failed in response to GET request to /.migration/doc/lock with no retries left, dispatching error... {}
[2019-11-11 11:17:06,549][INFO ][no.found.adminconsole.elasticsearch.IndexConfigurationActor] Waiting for admin cluster to become reachable ([Connection attempt to containerhost:9244 failed]). Retrying every [5 seconds]. {}
spray.can.Http$ConnectionAttemptFailedException: Connection attempt to containerhost:9244 failed
        at spray.can.client.HttpHostConnectionSlot$$anonfun$connecting$1.applyOrElse(HttpHostConnectionSlot.scala:87)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
        at spray.can.client.HttpHostConnectionSlot.aroundReceive(HttpHostConnectionSlot.scala:33)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

It is not possible to expand the admin console to the host. Other deployments are working.
And I cannot Login with its IP:12433.

On this hosts I have added IPtables rules:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 12191:12201 /* 900 profile::linux::firewall::deny_unused accept 12191-12201 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 12343 /* 900 profile::linux::firewall::deny_unused accept 12343 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 12443 /* 900 profile::linux::firewall::deny_unused accept 12443 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 12898:12908 /* 900 profile::linux::firewall::deny_unused accept 12898-12908 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 13898:13908 /* 900 profile::linux::firewall::deny_unused accept 13898-13908 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 18000:21999 /* 900 profile::linux::firewall::deny_unused accept 18000-21999 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 2112/* 900 profile::linux::firewall::deny_unused accept 2112 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports ssh /* 900 profile::linux::firewall::deny_unused accept 22 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 22191:22195 /* 900 profile::linux::firewall::deny_unused accept 22191-22195 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 9243 /* 900 profile::linux::firewall::deny_unused accept 9243 */
ACCEPT     tcp  --  anywhere             anywhere             multiport dports 9343/* 900 profile::linux::firewall::deny_unused accept 9343 */

Does anyone has experience with this?
Greetings
Malte

[Alex_Piggott]

There's 2 possible candidate issues here:

1. the 9244 described by the error is managed by a container called frc-services-forwarders-services-forwarder - is that container running / healthy?
2. Can you check that containerhost is mapped to the right IP address, eg docker -it frc-admin-consoles-admin-console host containerhost (and then check if 9244 is bound to the wrong host for some reason?)

[logger (Malte Leek)]

I checked it, but I think there is something more wrong in my Installation.

I will start a new one and try it step by step.

I found out there are a lot more errors. I will try a new one and if there are still errors I will come back.

Thanks
Malte