Essentially I'm trying to deploy an elk-stack into Kubernetes. I'm trying to learn Kubernetes so I'm doing it the hard way I've written my own manifests (kompose). I try to use the latest version my understanding that 18.14 .1. Trying to understand this automatic security deal and being that its Kubernetes to get the output of some of these auto-generated things I don't know if the logs for the pod will show that information.
So all my containers are running.
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-0 1/1 Running 0 19s
pod/flow-collector-75dc4b57f5-cgntl 1/1 Running 1 (20h ago) 44h
pod/kibana-b9958dbdd-sdtrh 1/1 Running 0 19s
pod/logstash-756db56455-tvbqv 1/1 Running 0 19s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch ClusterIP 10.107.238.92 <none> 9200/TCP,9300/TCP 19s
service/flow-collector LoadBalancer 10.101.108.62 192.168.0.178 9995:31490/TCP 2d5h
service/kibana LoadBalancer 10.96.173.147 192.168.0.176 5601:32611/TCP 19s
service/logstash LoadBalancer 10.97.242.50 192.168.0.177 5044:30537/TCP,9600:30214/TCP 19s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/flow-collector 1/1 1 1 2d5h
deployment.apps/kibana 1/1 1 1 19s
deployment.apps/logstash 1/1 1 1 19s
NAME DESIRED CURRENT READY AGE
replicaset.apps/flow-collector-5d588696f5 0 0 0 44h
replicaset.apps/flow-collector-67c95688df 0 0 0 2d5h
replicaset.apps/flow-collector-75dc4b57f5 1 1 1 44h
replicaset.apps/flow-collector-78944784c7 0 0 0 2d5h
replicaset.apps/kibana-b9958dbdd 1 1 1 19s
replicaset.apps/logstash-756db56455 1 1 1 19s
NAME READY AGE
statefulset.apps/elasticsearch 1/1 19s
When I go to Kibana's URL I get "Kibana server is not ready yet."
ok, logs
elasticsearch-0 logs
.name":"elasticsearch[elasticsearch-0][system_critical_read][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"khuA_9OXTRKB_6RqGV22AA","elasticsearch.node.id":"JETneWX1SUuUYv4_mdjx7A","elasticsearch.node.name":"elasticsearch-0","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2024-06-21T04:07:34.856Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-0][system_critical_read][T#2]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"khuA_9OXTRKB_6RqGV22AA","elasticsearch.node.id":"JETneWX1SUuUYv4_mdjx7A","elasticsearch.node.name":"elasticsearch-0","elasticsearch.cluster.name":"docker-cluster"}
kibana Logs
[2024-06-21T04:05:10.268+00:00][INFO ][plugins.assetManager] Server is NOT enabled
[2024-06-21T04:05:11.108+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 10.107.238.92:9200
[2024-06-21T04:05:12.865+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
[2024-06-21T04:05:13.210+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. write EPROTO 40D85DF5027F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:
My understanding is that it's supposed to be certs and things generated automatically they should be in the elastic containers config folder?
elasticsearch@elasticsearch-0:~/config$ ls -al
total 60
drwxrwxr-x 1 elasticsearch root 36 Jun 21 04:04 .
drwxrwxr-x 1 root root 46 Jun 21 04:04 ..
-rw-rw-r-- 1 root root 1042 Jun 10 23:42 elasticsearch-plugins.example.yml
-rw-rw---- 1 elasticsearch root 199 Jun 21 04:04 elasticsearch.keystore
-rw-rw-r-- 1 root root 53 Jun 10 23:42 elasticsearch.yml
-rw-rw-r-- 1 root root 2727 Jun 10 23:42 jvm.options
drwxrwxr-x 1 elasticsearch root 6 Jun 10 23:42 jvm.options.d
-rw-rw-r-- 1 root root 17969 Jun 10 23:42 log4j2.file.properties
-rw-rw-r-- 1 root root 12549 Jun 10 23:42 log4j2.properties
-rw-rw-r-- 1 root root 473 Jun 10 23:42 role_mapping.yml
-rw-rw-r-- 1 root root 197 Jun 10 23:42 roles.yml
-rw-rw-r-- 1 root root 0 Jun 10 23:42 users
-rw-rw-r-- 1 root root 0 Jun 10 23:42 users_roles
The only other thing I can think to mention is I'm using secrets in communities to program the environment variables.
apiVersion: v1
kind: Secret
metadata:
name: elastic-secret
type: Opaque
data:
stringData:
XPACK_SECURITY_ENABLED: 'true'
apiVersion: v1
kind: Secret
metadata:
name: kibana-secret
type: Opaque
data:
stringData:
ELASTICSEARCH_USERNAME: 'kibana_system'
ELASTICSEARCH_PASSWORD: 'XXXBASE64XXXX'
ELASTICSEARCH_HOSTS: 'https://elasticsearch:9200'
XPACK_SECURITY_ENABLED: 'true'
Thank you to anyone that's made it this far in the post I really appreciate it i'm happy to post my deployment files any other descriptions or information that might help and if I miss something completely obvious I apologize if I wasted anybody's time.