Trying to setup elasticsearch cluster with docker-compose

Hi,
My environment is two physical machine, both running in docker-compose.

I want to create elasticsearch cluster cross two docker container.

here's my configure:

ES1:
IP: 10.251.34.60

docker-compose.yml:

version: '2'
services:
  elasticsearch:
    image: elasticsearch:5.4.2
    hostname: es1
    container_name: es1
    user: elasticsearch
    ports:
    - 9200:9200
    - 9300:9300/udp
    environment:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
    command: /usr/share/elasticsearch/bin/elasticsearch

elasticsearch.yml:

http.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

cluster.name: prod_es_cluster
node.name: prod_es_node1
node.master: true
node.data: true
discovery.zen.ping_timeout: 10s

network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["10.251.34.50:9300","10.251.34.60:9300"]

ES2:
IP: 10.251.34.50

docker-compose.yml:

version: '2'
services:
  elasticsearch:
    image: elasticsearch:5.4.2
    hostname: es2
    container_name: es2
    user: elasticsearch
    ports:
    - 9200:9200
    - 9300:9300/udp
    environment:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
    command: /usr/share/elasticsearch/bin/elasticsearch

elasticsearch.yml:

http.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

cluster.name: prod_es_cluster
node.name: prod_es_node2
node.master: true
node.data: true
discovery.zen.ping_timeout: 10s

network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["10.251.34.50:9300","10.251.34.60:9300"]

how to setup cluster with docker container, any ideas?

Thanks! :grinning:

And here is logs, it seems like not connect each other

ES1: 

[2017-11-08T03:15:54,666][INFO ][o.e.n.Node ] [prod_es_node1] initializing ...
[2017-11-08T03:15:54,734][INFO ][o.e.e.NodeEnvironment ] [prod_es_node1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [9.
6gb], net total_space [9.9gb], spins? [unknown], types [rootfs]
[2017-11-08T03:15:54,734][INFO ][o.e.e.NodeEnvironment ] [prod_es_node1] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-11-08T03:15:54,736][INFO ][o.e.n.Node ] [prod_es_node1] node name [prod_es_node1], node ID [e7k9AdUIQkeCWIRifMOW9w]
[2017-11-08T03:15:54,736][INFO ][o.e.n.Node ] [prod_es_node1] version[5.4.2], pid[1], build[929b078/2017-06-15T02:29:28.122Z],
OS[Linux/3.10.0-327.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-11-08T03:15:54,736][INFO ][o.e.n.Node ] [prod_es_node1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSIn
itiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=tr
ue, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimiza
tion=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true,
-XX:+HeapDumpOnOutOfMemoryError, -Xms1g, -Xmx1g, -Des.path.home=/usr/share/elasticsearch]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [aggs-matrix-stats]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [ingest-common]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [lang-expression]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [lang-groovy]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [lang-mustache]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [lang-painless]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [percolator]
[2017-11-08T03:15:55,542][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [reindex]
[2017-11-08T03:15:55,543][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [transport-netty3]
[2017-11-08T03:15:55,543][INFO ][o.e.p.PluginsService ] [prod_es_node1] loaded module [transport-netty4]
[2017-11-08T03:15:55,543][INFO ][o.e.p.PluginsService ] [prod_es_node1] no plugins loaded
[2017-11-08T03:15:56,851][INFO ][o.e.d.DiscoveryModule ] [prod_es_node1] using discovery type [zen]
[2017-11-08T03:15:57,289][INFO ][o.e.n.Node ] [prod_es_node1] initialized
[2017-11-08T03:15:57,289][INFO ][o.e.n.Node ] [prod_es_node1] starting ...
[2017-11-08T03:15:57,419][INFO ][o.e.t.TransportService ] [prod_es_node1] publish_address {172.24.0.2:9300}, bound_addresses {[::]:9300}
[2017-11-08T03:15:57,425][INFO ][o.e.b.BootstrapChecks ] [prod_es_node1] bound or publishing to a non-loopback or non-link-local address,
enforcing bootstrap checks
[2017-11-08T03:16:07,490][INFO ][o.e.c.s.ClusterService ] [prod_es_node1] new_master {prod_es_node1}{e7k9AdUIQkeCWIRifMOW9w}{ja48giATT-m8HN
kUEDdBzQ}{172.24.0.2}{172.24.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-08T03:16:07,507][INFO ][o.e.h.n.Netty4HttpServerTransport] [prod_es_node1] publish_address {172.24.0.2:9200}, bound_addresses {[::]:
9200}
[2017-11-08T03:16:07,510][INFO ][o.e.n.Node ] [prod_es_node1] started
[2017-11-08T03:16:07,514][INFO ][o.e.g.GatewayService ] [prod_es_node1] recovered [0] indices into cluster_state

ES2:

[2017-11-08T03:16:25,279][INFO ][o.e.n.Node ] [prod_es_node2] initializing ...
[2017-11-08T03:16:25,345][INFO ][o.e.e.NodeEnvironment ] [prod_es_node2] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [9.
6gb], net total_space [9.9gb], spins? [unknown], types [rootfs]
[2017-11-08T03:16:25,345][INFO ][o.e.e.NodeEnvironment ] [prod_es_node2] heap size [990.7mb], compressed ordinary object pointers [true]
[2017-11-08T03:16:25,346][INFO ][o.e.n.Node ] [prod_es_node2] node name [prod_es_node2], node ID [KpS2TUq7Q5m2k1sONaD5zQ]
[2017-11-08T03:16:25,346][INFO ][o.e.n.Node ] [prod_es_node2] version[5.4.2], pid[1], build[929b078/2017-06-15T02:29:28.122Z],
OS[Linux/3.10.0-327.4.4.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-11-08T03:16:25,346][INFO ][o.e.n.Node ] [prod_es_node2] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSIn
itiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=tr
ue, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimiza
tion=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true,
-XX:+HeapDumpOnOutOfMemoryError, -Xms1g, -Xmx1g, -Des.path.home=/usr/share/elasticsearch]
[2017-11-08T03:16:26,100][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [aggs-matrix-stats]
[2017-11-08T03:16:26,100][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [ingest-common]
[2017-11-08T03:16:26,100][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [lang-expression]
[2017-11-08T03:16:26,100][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [lang-groovy]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [lang-mustache]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [lang-painless]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [percolator]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [reindex]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [transport-netty3]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] loaded module [transport-netty4]
[2017-11-08T03:16:26,101][INFO ][o.e.p.PluginsService ] [prod_es_node2] no plugins loaded
[2017-11-08T03:16:27,379][INFO ][o.e.d.DiscoveryModule ] [prod_es_node2] using discovery type [zen]
[2017-11-08T03:16:27,811][INFO ][o.e.n.Node ] [prod_es_node2] initialized
[2017-11-08T03:16:27,811][INFO ][o.e.n.Node ] [prod_es_node2] starting ...
[2017-11-08T03:16:27,940][INFO ][o.e.t.TransportService ] [prod_es_node2] publish_address {172.21.0.2:9300}, bound_addresses {[::]:9300}
[2017-11-08T03:16:27,946][INFO ][o.e.b.BootstrapChecks ] [prod_es_node2] bound or publishing to a non-loopback or non-link-local address,
enforcing bootstrap checks
[2017-11-08T03:16:37,986][INFO ][o.e.c.s.ClusterService ] [prod_es_node2] new_master {prod_es_node2}{KpS2TUq7Q5m2k1sONaD5zQ}{RtjqOsnjTlSvg4
sjqoJWJQ}{172.21.0.2}{172.21.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-08T03:16:38,006][INFO ][o.e.h.n.Netty4HttpServerTransport] [prod_es_node2] publish_address {172.21.0.2:9200}, bound_addresses {[::]:
9200}
[2017-11-08T03:16:38,009][INFO ][o.e.n.Node ] [prod_es_node2] started
[2017-11-08T03:16:38,016][INFO ][o.e.g.GatewayService ] [prod_es_node2] recovered [0] indices into cluster_state

Does https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-prod-cluster-composefile help?

Hi warkolm,

The example looks like in same physical machine, running one docker compose file and create two container.

Networking should be fine.

But in my case, like this

How to configure the networking, that elasticsearch cluster can be connected each other.

Thank you :smiley:

Ok, let's look at docker-compose.yml:

version: '2'
services:
  elasticsearch:
    image: elasticsearch:5.4.2
    hostname: es2
    container_name: es2
    user: elasticsearch
    ports:
    - 9200:9200
    - 9300:9300/udp
    environment:
      ES_JAVA_OPTS: "-Xms1g -Xmx1g"
    command: /usr/share/elasticsearch/bin/elasticsearch

The main problem with this is - 9300:9300/udp. Elasticsearch discovery uses TCP and you are only allowing UDP. Removed /udp and you should get everything working.

A few other tips for your elasticsearch.yml, all you need is:

cluster.name: prod_es_cluster
node.name: prod_es_node2
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["10.251.34.50","10.251.34.60"]

Everything else you have defined is the defaults, so you don't need to provide them.

Also discovery.zen.minimum_master_nodes: 1 is bad, see https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#_minimum_master_nodes

Finally, you should 100% be using our docker image here - https://github.com/elastic/elasticsearch-docker. We keep it up to date and apply evolving best practises :slight_smile:

Hi warkolm,

Now I removed /udp config.

docker image is using elasticsearch:5.4.2 but export from other testing docker machine, then import this server

so named es:542

docker-compose.yml like this

version: '2'

services:
  elasticsearch:
	image: es:542
    hostname: es2
    container_name: es2
    user: elasticsearch
    ports:
    - 9200:9200
    - 9300:9300
    command: /usr/share/elasticsearch/bin/elasticsearch
    environment:
    - ES_JAVA_OPTS=-Xms1g -Xmx1g

change elasticsearch.yml like this discovery.zen.minimum_master_nodes: 2

http.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 2

cluster.name: prod_es_cluster
node.name: prod_es_node1
node.master: true
node.data: true
discovery.zen.ping_timeout: 10s

network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["127.0.0.1", "[::1]"]

and the log:

[2017-11-09T03:49:09,261][INFO ][o.e.t.TransportService ] [prod_es_node1] publish_address {172.24.0.2:9300}, bound_addresses {[::]:9300}
[2017-11-09T03:49:09,270][INFO ][o.e.b.BootstrapChecks ] [prod_es_node1] bound or publishing to a non-loopback or non-link-local address,e
nforcing bootstrap checks
[2017-11-09T03:49:19,319][WARN ][o.e.d.z.ZenDiscovery ] [prod_es_node1] not enough master nodes discovered during pinging (found [[Candid
ate{node={prod_es_node1}{BxKzhOnJTUC50cYTz_HmfA}{nISnloDKQBul5sVdPd5m_Q}{172.24.0.2}{172.24.0.2:9300}, clusterStateVersion=-1}]], but needed[
2]), pinging again
[2017-11-09T03:49:29,321][WARN ][o.e.d.z.ZenDiscovery ] [prod_es_node1] not enough master nodes discovered during pinging (found [[Candid
ate{node={prod_es_node1}{BxKzhOnJTUC50cYTz_HmfA}{nISnloDKQBul5sVdPd5m_Q}{172.24.0.2}{172.24.0.2:9300}, clusterStateVersion=-1}]], but needed[
2]), pinging again
[2017-11-09T03:49:39,297][WARN ][o.e.n.Node ] [prod_es_node1] timed out while waiting for initial discovery state - timeout: 30
s
[2017-11-09T03:49:39,307][INFO ][o.e.h.n.Netty4HttpServerTransport] [prod_es_node1] publish_address {172.24.0.2:9200}, bound_addresses {[::]:
9200}
[2017-11-09T03:49:39,310][INFO ][o.e.n.Node ] [prod_es_node1] started
[2017-11-09T03:49:39,323][WARN ][o.e.d.z.ZenDiscovery ] [prod_es_node1] not enough master nodes discovered during pinging (found [[Candid
ate{node={prod_es_node1}{BxKzhOnJTUC50cYTz_HmfA}{nISnloDKQBul5sVdPd5m_Q}{172.24.0.2}{172.24.0.2:9300}, clusterStateVersion=-1}]], but needed[
2]), pinging again

it still doesn't connected each other :disappointed_relieved:

That's only ever going to connect to itself.

now elasticsearh.yml is

http.host: 0.0.0.0
transport.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 2

cluster.name: prod_es_cluster
node.name: prod_es_node1
node.master: true
node.data: true
discovery.zen.ping_timeout: 10s

network.host: 0.0.0.0
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["10.251.34.50","10.251.34.60"]

and logs

ES1

[2017-11-09T05:56:10,552][INFO ][o.e.t.TransportService ] [prod_es_node1] publish_address {172.24.0.2:9300}, bound_addresses {[::]:9300}
[2017-11-09T05:56:10,558][INFO ][o.e.b.BootstrapChecks ] [prod_es_node1] bound or publishing to a non-loopback or non-link-local address,enforcing bootstrap checks
[2017-11-09T05:56:40,576][WARN ][o.e.n.Node ][prod_es_node1] timed out while waiting for initial discovery state - timeout: 30s
[2017-11-09T05:56:40,584][INFO ][o.e.h.n.Netty4HttpServerTransport] [prod_es_node1] publish_address {172.24.0.2:9200}, bound_addresses {[::]:9200}
[2017-11-09T05:56:40,587][INFO ][o.e.n.Node ] [prod_es_node1] started

ES2

[2017-11-09T09:37:20,084][WARN ][o.e.d.z.ZenDiscovery ] [prod_es_node2]failed to connect to master [{prod_es_node1}{BxKzhOnJTUC50cYTz_Hm fA}{zqtU07jfQJOrmB9AYL01Ig}{172.24.0.2}{172.24.0.2:9300}], retrying...
org.elasticsearch.transport.ConnectTransportException:[prod_es_node1][172.24.0.2:9300] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:361) ~[?:?]
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:549) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:315) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:302) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.discovery.zen.ZenDiscovery.joinElectedMaster(ZenDiscovery.java:468) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:420) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.discovery.zen.ZenDiscovery.access$4100(ZenDiscovery.java:83) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.discovery.zen.ZenDiscovery$JoinThreadControl$1.run(ZenDiscovery.java:1197) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.4.2.
jar:5.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException:Connection refused: 172.24.0.2/172.24.0.2:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]

Now I use openvswitch and pipework to build cluster network

ovs setting:

#ES1
sudo brctl addbr br0
sudo ip link set dev br0 up
sudo ovs-vsctl add-br ovs0
sudo ovs-vsctl set bridge ovs0 stp_enable=true
sudo ovs-vsctl add-port ovs0 br0
sudo ovs-vsctl add-port ovs0 gre0 -- set interface gre0 type=gre options:remote_ip=10.251.34.50

pipwork setting:

sudo pipework-master/pipework br0 -i eth1 es1 172.28.0.2/8

docker container network information:

elasticsearch@es1:/usr/share/elasticsearch$ ip a

521: eth1@if522: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 4e:06:91:91:1f:3e brd ff:ff:ff:ff:ff:ff
inet 172.28.0.2/8 brd 172.255.255.255 scope global eth1
valid_lft forever preferred_lft forever

after setting both two docker container, cluster should be work !!

ES1

elasticsearch@es1:/usr/share/elasticsearch$ ping 172.18.0.2
PING 172.18.0.2 (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: icmp_seq=0 ttl=64 time=1.222 ms
64 bytes from 172.18.0.2: icmp_seq=1 ttl=64 time=0.339 ms
64 bytes from 172.18.0.2: icmp_seq=2 ttl=64 time=0.547 ms
64 bytes from 172.18.0.2: icmp_seq=3 ttl=64 time=0.303 ms
64 bytes from 172.18.0.2: icmp_seq=4 ttl=64 time=0.333 ms
64 bytes from 172.18.0.2: icmp_seq=5 ttl=64 time=0.362 ms

#ES logs:
[2017-11-17T02:06:21,205][INFO ][o.e.n.Node ] [prod_es_node1] started
[2017-11-17T02:06:21,223][INFO ][o.e.c.s.ClusterService ] [prod_es_node1] new_master {prod_es_node1}{IoG7C5BYQ26AYsggMuJo2A}{AL3oD0itR1CgibIr-x-8Sg}{172.28.0.2}{172.28.0.2:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-17T02:06:21,254][INFO ][o.e.g.GatewayService ] [prod_es_node1] recovered [0] indices into cluster_state
[2017-11-17T02:06:29,205][INFO ][o.e.c.s.ClusterService ] [prod_es_node1] added {{prod_es_node2}{ZA7CwKyBQS2gm-OamVSz2g}{U6WQyzwoQCiDkwBvmUxoQw}{172.18.0.2}{172.18.0.2:9300},}, reason: zen-disco-node-join[{prod_es_node2}{ZA7CwKyBQS2gm-OamVSz2g}{U6WQyzwoQCiDkwBvmUxoQw}{172.18.0.2}{172.18.0.2:9300}]

curl localhost:9200/_cluster/health?pretty
{
"cluster_name" : "prod_es_cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.