Could not get cluster status after master node goes down

Elastic Stack Elasticsearch
41 
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
[2018-06-07T09:45:01,773][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [172.22.107.22:10000] failed to execute on node [b1rXMHeJTh2LfTN1klzHzA]
org.elasticsearch.transport.NodeNotConnectedException: [172.22.107.21:10000][172.22.107.21:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.7.jar:5.6.7]
[2018-06-07T09:45:01,773][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [172.22.107.22:10000] failed to execute on node [b1rXMHeJTh2LfTN1klzHzA]
org.elasticsearch.transport.NodeNotConnectedException: [172.22.107.21:10000][172.22.107.21:9300] Node not connected
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.rest.action.admin.cluster.RestNodesStatsAction.lambda$prepareRequest$13(RestNodesStatsAction.java:189) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:262) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:200) [elasticsearch-5.6.7.jar:5.6.7]

all the posted logs belong to new master node (172.22.107.22).

PS. I think it would be nice to have the possiblilty to attach the .txt files to the posts. Sometimes it's not easy to share the files...

42

The following lines are the logs for the node 172.22.107.20:

[2018-06-07T09:42:29,407][INFO ][o.e.n.Node               ] [172.22.107.20:10000] initializing ...
[2018-06-07T09:42:29,556][INFO ][o.e.e.NodeEnvironment    ] [172.22.107.20:10000] using [1] data paths, mounts [[/opt (/dev/sdb)]], net usable_space [112.7gb], net total_space [150gb], spins? [possibly], types
[btrfs]
[2018-06-07T09:42:29,556][INFO ][o.e.e.NodeEnvironment    ] [172.22.107.20:10000] heap size [11.9gb], compressed ordinary object pointers [true]
[2018-06-07T09:42:29,692][INFO ][o.e.n.Node               ] [172.22.107.20:10000] node name [172.22.107.20:10000], node ID [w49SCPmcRvSsW0_948B_rA]
[2018-06-07T09:42:29,693][INFO ][o.e.n.Node               ] [172.22.107.20:10000] version[5.6.7], pid[24151], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.21.1.el7.x86_64/amd64], JVM[Oracle Corp
oration/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12]
[2018-06-07T09:42:29,693][INFO ][o.e.n.Node               ] [172.22.107.20:10000] JVM arguments [-Xms12g, -Xmx12g, -Xss1m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOc
cupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKey
SetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/opt/bdm
_current//log/es_heapdump_20180607-0942.hprof, -Dlog4j2.disable.jmx=true, -Delasticsearch, -Des.pidfile=/opt/bdm_current//pid/172.22.107.20:10000/elasticsearch.pid, -Des.path.home=/opt/bdm_current/, -Des.path.l
ogs=/opt/bdm_current//log, -Des.path.data=/opt/bdm4_data, -Des.path.work=/tmp/elasticsearch/172.22.107.20:10000]
[2018-06-07T09:42:30,982][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [aggs-matrix-stats]
[2018-06-07T09:42:30,982][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [ingest-common]
[2018-06-07T09:42:30,982][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [lang-expression]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [lang-groovy]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [lang-mustache]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [lang-painless]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [parent-join]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [percolator]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [reindex]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [transport-netty3]
[2018-06-07T09:42:30,983][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] loaded module [transport-netty4]
[2018-06-07T09:42:30,984][INFO ][o.e.p.PluginsService     ] [172.22.107.20:10000] no plugins loaded
[2018-06-07T09:42:33,220][INFO ][o.e.d.DiscoveryModule    ] [172.22.107.20:10000] using discovery type [zen]
[2018-06-07T09:42:34,321][INFO ][o.e.n.Node               ] [172.22.107.20:10000] initialized
[2018-06-07T09:42:34,322][INFO ][o.e.n.Node               ] [172.22.107.20:10000] starting ...
[2018-06-07T09:42:34,529][INFO ][o.e.t.TransportService   ] [172.22.107.20:10000] publish_address {172.22.107.20:9300}, bound_addresses {[::]:9300}
[2018-06-07T09:42:34,541][INFO ][o.e.b.BootstrapChecks    ] [172.22.107.20:10000] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-06-07T09:42:37,849][INFO ][o.e.c.s.ClusterService   ] [172.22.107.20:10000] detected_master {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300}, added {{172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300},{172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300},}, reason: zen-disco-receive(from master [master {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300} committed version [45]])
[2018-06-07T09:42:39,887][INFO ][o.e.h.n.Netty4HttpServerTransport] [172.22.107.20:10000] publish_address {172.22.107.20:9200}, bound_addresses {[::]:9200}
[2018-06-07T09:42:39,887][INFO ][o.e.n.Node               ] [172.22.107.20:10000] started
[2018-06-07T09:44:57,366][INFO ][o.e.d.z.ZenDiscovery     ] [172.22.107.20:10000] master_left [{172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300}], reason [shut_down]
[2018-06-07T09:44:57,369][WARN ][o.e.d.z.ZenDiscovery     ] [172.22.107.20:10000] master left (reason = shut_down), current nodes: nodes:
   {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300}, master
   {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300}
   {172.22.107.20:10000}{w49SCPmcRvSsW0_948B_rA}{VnlkmgbtQuO793YmrfYDnA}{172.22.107.20}{172.22.107.20:9300}, local
43 
[2018-06-07T09:44:58,036][DEBUG][o.e.a.a.c.s.TransportClusterStateAction] [172.22.107.20:10000] no known master node, scheduling a retry
[2018-06-07T09:44:58,133][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [172.22.107.20:10000] no known master node, scheduling a retry
[2018-06-07T09:45:00,448][INFO ][o.e.c.s.ClusterService   ] [172.22.107.20:10000] detected_master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300}, reason: zen-disco-receive(from master [master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300} committed version [76]])
[2018-06-07T09:45:00,458][WARN ][o.e.c.NodeConnectionsService] [172.22.107.20:10000] failed to connect to node {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [172.22.107.21:10000][172.22.107.21:9300] connect_timeout[30s]
        at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?]
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.7.jar:5.6.7]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 172.22.107.21/172.22.107.21:9300
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
        ... 1 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]
        ... 1 more
[2018-06-07T09:45:00,828][INFO ][o.e.c.s.ClusterService   ] [172.22.107.20:10000] removed {{172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300},}, reason: zen-disco-receive(from master [master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300} committed version [77]])
(END)
44

Upload to gist.github.com if you wish.

1 Like
45

On which machine did you run then:

curl -XGET localhost:9200/_cluster/health?pretty

?

46

On both.

I'm also using head plugin which is connected to 172.22.107.22 using port forwarding. After stopping the master it doesn't respond as well.

47

I believe you have something wrong in your network configuration. You really need to check your firewall rules. This kind of problem reported on this forum is caused by this 99% of the time.

48

After many tests I found that this problem doesn't occur if I start elasticsearch manually with the java command and same arguments. So it could not be a problem with network or hardware, IMHO.
We have a java application which read the jvm configuration from a xml file and start ES process using

		ProcessBuilder pb = new ProcessBuilder(command);
		pb = pb.redirectErrorStream(true);
		pb.start()

It seems that something goes wrong when a nodes crashes.
I've started ES instances manually using the following command (not using our java application to start it) and tested the cluster HA and everything was just fine :

sudo -u bdm java -Xms12g -Xmx12g -Xss1m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/bdm_current//log/es_heapdump_20180608-0741.hprof -Dlog4j2.disable.jmx=true -Delasticsearch -Des.pidfile=/opt/bdm_current//pid/172.22.107.20:10000/elasticsearch.pid -Des.path.home=/opt/bdm_current/ -Des.path.logs=/opt/bdm_current//log -Des.path.data=/opt/bdm4_data -Des.path.work=/tmp/elasticsearch/172.22.107.20:10000 -cp /opt/bdm_current//es-lib/*:/opt/bdm_current//modules org.elasticsearch.bootstrap.Elasticsearch

these command copied from the logs in parent java process, so the same parameters will be used to start ES.

49

We are not supporting launching elasticsearch from another process.

I don't know what you can do here.

But why are you doing this and not starting elasticsearch as you would start MySQL for example?

1 Like
50

we wanted to simplify the work of our customers not to deal with many configuration files. It is easier for them to have all the configurations of the application(s) in a single file....

51

Hi David,

I've changed the way of starting ES by using the scripts, however the ES start script get executed by the parent script which start all the processes, so no extra works for the system admins as well.

After that HA working just fine.

Thank you for your great advices.

Cheers,
Vahid

52

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.