Hi Team,
We are trying to configure remote cluster with ECK and facing the below issue
cluster1 (Master) - Managed by ECK which has es, kibana & fluentd
cluster2 (Remote) - Managed by ECK which has es & fluentd
- In cluster1, Execute the eck.yml & configured fluentd. After this step, we are able to get indices by (curl -u elastic:XXXXXXXXX -k "https://elasticsearch-es-http:9200/_cat/indices)
eck.yml:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
namespace: es-test
labels:
app: elasticsearch
spec:
version: 7.9.3
nodeSets:
- name: default
count: 3
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
http:
service:
spec:
type: LoadBalancer
ports:
- port: 9200
targetPort: 9200
protocol: TCP
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
namespace: es-test
spec:
version: 7.9.3
count: 1
elasticsearchRef:
name: elasticsearch
http:
tls:
selfSignedCertificate:
disabled: true
- In cluster2, Same step1 is followed except kibana.
- In cluster2, we have exposed svc with transport port 9300.
es-svc.yml:
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-logging
namespace: "es-test"
labels:
app: elasticsearch
spec:
selector:
common.k8s.elastic.co/type: elasticsearch
elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
cluster1:
[root@k8s-master01 eck]# kubectl get po -n es-test
NAME READY STATUS RESTARTS AGE
elasticsearch-es-default-0 1/1 Running 0 97m
elasticsearch-es-default-1 1/1 Running 0 98m
elasticsearch-es-default-2 1/1 Running 0 100m
fluentd-8bb4q 1/1 Running 0 150m
fluentd-h8j6m 1/1 Running 0 150m
fluentd-h8lst 1/1 Running 0 150m
fluentd-p9j2g 1/1 Running 0 150m
kibana-kb-55c7584fd6-r62lc 1/1 Running 0 107m
[root@k8s-master01 eck]# kubectl get svc -n es-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-es-default ClusterIP None <none> <none> 5h5m
elasticsearch-es-http LoadBalancer x.x.x.x <pending> 9200:31300/TCP 5h5m
kibana-kb-http ClusterIP x.x.x.x <none> 5601/TCP 107m
cluster2:
[root@k8s-master01 eck]# kubectl get po -n es-test
NAME READY STATUS RESTARTS AGE
elasticsearch-es-default-0 1/1 Running 0 97m
elasticsearch-es-default-1 1/1 Running 0 98m
elasticsearch-es-default-2 1/1 Running 0 100m
fluentd-8bb4q 1/1 Running 0 150m
fluentd-h8j6m 1/1 Running 0 150m
fluentd-h8lst 1/1 Running 0 150m
fluentd-p9j2g 1/1 Running 0 150m
[root@k8s-master01 eck]# kubectl get svc -n es-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch-es-default ClusterIP None <none> <none> 5h5m
elasticsearch-es-http LoadBalancer x.x.x.x <pending> 9200:31300/TCP 5h5m
elasticsearch-logging ClusterIP x.x.x.x <none> 9200/TCP,9300/TCP 9h
kibana-kb-http ClusterIP x.x.x.x <none> 5601/TCP 107m
- Copied remote.ca.crt from cluster2 to cluster1 and created secret in cluster1
kubectl get secret elasticsearch-es-transport-certs-public -n es-test -o go-template='{{index .data "ca.crt" | base64decode}}' > remote.ca.crt
kubectl create secret generic remote-certs --from-file=remote.ca.crt -n es-test
-- Same has been done from cluster1 to cluster2.
- In both cluster, updated elasticsearch with remote.ca.crt
nodeSets:
- config:
xpack.security.transport.ssl.certificate_authorities:
- /usr/share/elasticsearch/config/other/remote.ca.crt
name: default
count: 3
podTemplate:
spec:
containers:
- name: elasticsearch
volumeMounts:
- mountPath: /usr/share/elasticsearch/config/other
name: remote-certs
volumes:
- name: remote-certs
secret:
secretName: remote-certs
-
In cluster2, created virtual service for elasticsearch-logging.es-test:9300 with nodeip 1.1.1.1:9300
-
In cluster1 kibana, remote cluster config
{
"persistent": {
"cluster": {
"remote": {
"cluster-test": {
"mode": "proxy",
"proxy_address": "1.1.1.1:9300"
}
}
}
}
}
We are getting below error in cluster1 and cluster2:
ES logs in cluster1:
Java exception with signature check failed and PKIX path validation failed
ES logs in cluster2:
{"type": "server", "timestamp": "2021-04-21T06:36:16,892Z", "level": "WARN", "component": "o.e.x.c.s.t.n.SecurityNetty4Transport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-es-default-2", "message": "client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/x.x.x.x:9300, remoteAddress=/x.x.x.x:21989}", "cluster.uuid": "cEJL5C68Sqy7ZhT5yh3VSA", "node.id": "fBD_2ISDS0CIsDbzGXxMpw" }