Hi All,
I have checked numerous forums but haven't found a solution, so I am raising this question here because of my unique setup. I have Elasticsearch and Kibana running on a Proxmox server inside containers. My entire Proxmox server is behind NAT with a public IP address.
I am using Nginx as a reverse proxy to map all my services to a domain. Here are the details of my setup:
- Proxmox server IP: 276.89.76.xx
- Nginx container IP: 192.168.13.107
- Elasticsearch container IP: 192.168.13.139
- Kibana and Fleet server IP: 192.168.13.152
My Installation steps of elastic
# After Installing the elastic search using official guide
cd /usr/share/elasticsearch/
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
export ES_PATH_CONF=/etc/elasticsearch/
nano $ES_PATH_CONF/elasticsearch.yml
./bin/elasticsearch-certutil http
unzip elasticsearch-ssl-http.zip
cp /usr/share/elasticsearch/elasticsearch/http.p12 /etc/elasticsearch/
nano $ES_PATH_CONF/elasticsearch.yml
# I have no password in the certificate that I have generated
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
nano /etc/elasticsearch/elasticsearch.yml
# I am getting memory lock error so I have fixed using the below
ulimit -n 65536
ulimit -u 2048
nano /etc/security/limits.conf
nano /etc/sysctl.conf
Added elasticsearch-ca.pem to kibana server
cd /usr/share/elasticsearch/kibana
ls
cat elasticsearch-ca.pem
cd /etc/kibana
nano elasticsearch-ca.pem
nano kibana.yml
Nginx Server domain mapping -
server {
server_name kibana.domain.app;
location / {
proxy_pass https://192.168.13.152:5601/;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 900s;
}
}
elasticsearch.yml file
cluster.name: elastic_techs
node.name: elastic
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
client_authentication: required
keystore.path: elastic-certificates.p12
truststore.path: elastic-certificates.p12
cluster.initial_master_nodes: ["elastic"]
kibana.yml
server.port: 5601
server.host: 0.0.0.0
elasticsearch.hosts: ['https://192.168.13.139:9200']
logging.appenders.file.type: file
logging.appenders.file.fileName: /var/log/kibana/kibana.log
logging.appenders.file.layout.type: json
logging.root.appenders: [default, file]
pid.file: /run/kibana/kibana.pid
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem
elasticsearch.username: "kibana_system"
elasticsearch.password: "Some Random Text Based Passoword"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana-server.crt
server.ssl.key: /etc/kibana/kibana-server.key
Installed fleet server on the same container as kibana
Telnet Results below when I tried to connect to proxmox server public IP from a external machine
root@imaging:~/elastic-agent-8.13.4-linux-x86_64# telnet 276.89.76.xx 9200
Trying 276.89.76.xx...
Connected to 276.89.76.xx.
Escape character is '^]'.
I am enrolling the fleet agent using the command without insecure I am not able setup the agent
sudo ./elastic-agent install --url=https://kibana.domain.app:8220 --enrollment-token=token==
Getting the below error
root@imaging:~/elastic-agent-8.13.4-linux-x86_64# sudo ./elastic-agent install --url=https://kibana.domain.app:8220 --enrollment-token=token==
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[= ] Service Started [11s] Elastic Agent successfully installed, starting enrollment.
[=== ] Waiting For Enroll... [12s] {"log.level":"info","@timestamp":"2024-07-26T12:21:38.774Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://kibana.domain.app:8220/","ecs.version":"1.6.0"}
[ ==] Waiting For Enroll... [13s] {"log.level":"info","@timestamp":"2024-07-26T12:21:39.294Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":528},"message":"1st enrollment attempt failed, retrying for 10m0s, every 1m0s enrolling to URL: https://kibana.domain.app:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html
[== ] Uninstalled [14s] Error uninstalling. Printing logs
.
.
{debug 2024-07-26 12:21:39.729530731 +0000 UTC m=+15.751428623 Loaded configuration from /root/elastic-agent-8.13.4-linux-x86_64/elastic-agent.yml github.com/elastic/elastic-agent/internal/pkg/config/loader.go:45 }
{debug 2024-07-26 12:21:39.729551801 +0000 UTC m=+15.751449692 Merged configuration from /root/elastic-agent-8.13.4-linux-x86_64/elastic-agent.yml into result github.com/elastic/elastic-agent/internal/pkg/config/loader.go:57 }
{debug 2024-07-26 12:21:39.729556839 +0000 UTC m=+15.751454731 Merged all configuration files from [/root/elastic-agent-8.13.4-linux-x86_64/elastic-agent.yml], no external input files github.com/elastic/elastic-agent/internal/pkg/config/loader.go:64 }
{debug 2024-07-26 12:21:39.729698729 +0000 UTC m=+15.751596620 composable Starting controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:118 }
{debug 2024-07-26 12:21:39.729714277 +0000 UTC m=+15.751612169 composable Started controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:161 }
{debug 2024-07-26 12:21:39.729805537 +0000 UTC m=+15.751703429 composable Kubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go:53 }
{debug 2024-07-26 12:21:39.72981408 +0000 UTC m=+15.751711976 composable kubernetes_secrets provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets.go:81 }
{debug 2024-07-26 12:21:39.729831154 +0000 UTC m=+15.751729051 composable Variable state changed for composable inputs; debounce started github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:197 }
{debug 2024-07-26 12:21:39.729861325 +0000 UTC m=+15.751759218 composable.providers.kubernetes Kubernetes provider for resource pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes/kubernetes.go:106 }
{debug 2024-07-26 12:21:39.729857153 +0000 UTC m=+15.751755050 docker Docker client will negotiate the API version on the first request. github.com/elastic/elastic-agent-autodiscover@v0.6.7/docker/client.go:49 }
{debug 2024-07-26 12:21:39.729896172 +0000 UTC m=+15.751794065 composable.providers.kubernetes Kubernetes provider for resource node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes/kubernetes.go:106 }
{info 2024-07-26 12:21:39.730076524 +0000 UTC m=+15.751974416 composable.providers.docker Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker/docker.go:44 }
{debug 2024-07-26 12:21:39.830392957 +0000 UTC m=+15.852290922 composable Computing new variable state for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:213 }
{debug 2024-07-26 12:21:39.830548547 +0000 UTC m=+15.852446495 composable Stopping controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:165 }
{debug 2024-07-26 12:21:39.930361552 +0000 UTC m=+15.952259497 composable Stopped controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:194 }
Error: enroll command failed for unknown reason: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html
If I try to enroll with --insecure I am able to install the agent
root@coolr-imaging:~/elastic-agent-8.13.4-linux-x86_64# sudo ./elastic-agent install --url=https://kibana.domain.app:8220 --enrollment-token=token-here --insecure
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[ ==] Service Started [5s] Elastic Agent successfully installed, starting enrollment.
[ ===] Waiting For Enroll... [5s] {"log.level":"warn","@timestamp":"2024-07-26T12:44:25.401Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[ ==] Waiting For Enroll... [5s] {"log.level":"info","@timestamp":"2024-07-26T12:44:26.183Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://kibana.stream4tech.app:8220/","ecs.version":"1.6.0"}
[====] Waiting For Enroll... [6s] {"log.level":"warn","@timestamp":"2024-07-26T12:44:26.396Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[== ] Waiting For Enroll... [7s] {"log.level":"info","@timestamp":"2024-07-26T12:44:27.405Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-07-26T12:44:27.407Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":300},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[= ] Done [7s]
Elastic Agent has been successfully installed.
Agent is showing healthy but there no metrics
This the status showing on kibana dashboard Below are the machines that enrolled inside proxmox server using private IP
There is a thing like if I enroll a machine in fleet server internally and externally I am able to enroll
root@sftp-server:~/elastic-agent-8.13.4-linux-x86_64# sudo ./elastic-agent install --url=https://192.168.13.168:8220 --enrollment-token=token-here --insecure
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:
[====] Service Started [5s] Elastic Agent successfully installed, starting enrollment.
[=== ] Waiting For Enroll... [5s] {"log.level":"warn","@timestamp":"2024-07-26T12:33:43.476Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[= ] Waiting For Enroll... [5s] {"log.level":"info","@timestamp":"2024-07-26T12:33:43.649Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://192.168.13.168:8220/","ecs.version":"1.6.0"}
[ =] Waiting For Enroll... [5s] {"log.level":"warn","@timestamp":"2024-07-26T12:33:43.868Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[ =] Waiting For Enroll... [6s] {"log.level":"info","@timestamp":"2024-07-26T12:33:44.641Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
[ =] Waiting For Enroll... [6s] {"log.level":"info","@timestamp":"2024-07-26T12:33:44.688Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":300},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[ =] Done [6s]
Elastic Agent has been successfully installed.
root@sftp-server:~/elastic-agent-8.13.4-linux-x86_64#
Agent Policy under I am assigning new agents
Fleet Server Policy - ROHAN
rev. 14
Below is the logs that I am getting from elastic agent
ISSUE - I am not able to get metrics from the machines that enrolled outside the network (proxmox) what is the certificate verificate part I am missing? and How can I fix