# I am not able to get metrics from the machines that enrolled outside the network

**URL:** <https://discuss.elastic.co/t/i-am-not-able-to-get-metrics-from-the-machines-that-enrolled-outside-the-network/363862>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring, elastic-stack-security, docker\
**Created:** [July 26, 2024, 12:54pm UTC](https://discuss.elastic.co/t/i-am-not-able-to-get-metrics-from-the-machines-that-enrolled-outside-the-network/363862 "2024-07-26T12:54:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rohan\_sirohi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohan_sirohi/32/136323_2.png) [@Rohan\_sirohi](https://discuss.elastic.co/u/Rohan_sirohi)\
**Post date:** [July 26, 2024, 12:54pm UTC](https://discuss.elastic.co/t/i-am-not-able-to-get-metrics-from-the-machines-that-enrolled-outside-the-network/363862/1 "2024-07-26T12:54:26Z")

</div>

Hi All,

I have checked numerous forums but haven't found a solution, so I am raising this question here because of my unique setup. I have Elasticsearch and Kibana running on a Proxmox server inside containers. My entire Proxmox server is behind NAT with a public IP address.

I am using Nginx as a reverse proxy to map all my services to a domain. Here are the details of my setup:

- Proxmox server IP: 276.89.76.xx
- Nginx container IP: 192.168.13.107
- Elasticsearch container IP: 192.168.13.139
- Kibana and Fleet server IP: 192.168.13.152

**My Installation steps of elastic**

```auto
# After Installing the elastic search using official guide
cd /usr/share/elasticsearch/
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
export ES_PATH_CONF=/etc/elasticsearch/
nano $ES_PATH_CONF/elasticsearch.yml
./bin/elasticsearch-certutil http
unzip elasticsearch-ssl-http.zip
cp /usr/share/elasticsearch/elasticsearch/http.p12 /etc/elasticsearch/
nano $ES_PATH_CONF/elasticsearch.yml
# I have no password in the certificate that I have generated
./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
nano /etc/elasticsearch/elasticsearch.yml
# I am getting memory lock error so I have fixed using the below
ulimit -n 65536 
ulimit -u 2048
nano /etc/security/limits.conf
nano /etc/sysctl.conf

```

Added elasticsearch-ca.pem to kibana server

```auto
cd /usr/share/elasticsearch/kibana
ls
cat elasticsearch-ca.pem
cd /etc/kibana
nano elasticsearch-ca.pem
nano kibana.yml

```

Nginx Server domain mapping -

```auto
server {
server_name kibana.domain.app;

    location / {
        proxy_pass https://192.168.13.152:5601/;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
	proxy_read_timeout 900s;
    }

}

```

elasticsearch.yml file

```auto
cluster.name: elastic_techs
node.name: elastic
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: required
  keystore.path: elastic-certificates.p12
  truststore.path: elastic-certificates.p12
cluster.initial_master_nodes: ["elastic"]

```

kibana.yml

```auto
server.port: 5601
server.host: 0.0.0.0
elasticsearch.hosts: ['https://192.168.13.139:9200']
logging.appenders.file.type: file
logging.appenders.file.fileName: /var/log/kibana/kibana.log
logging.appenders.file.layout.type: json
logging.root.appenders: [default, file]
pid.file: /run/kibana/kibana.pid
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: /etc/kibana/elasticsearch-ca.pem
elasticsearch.username: "kibana_system"
elasticsearch.password: "Some Random Text Based Passoword"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/kibana-server.crt
server.ssl.key: /etc/kibana/kibana-server.key

```

Installed fleet server on the same container as kibana

Telnet Results below when I tried to connect to proxmox server public IP from a external machine

```auto
root@imaging:~/elastic-agent-8.13.4-linux-x86_64# telnet 276.89.76.xx 9200
Trying 276.89.76.xx...
Connected to 276.89.76.xx.
Escape character is '^]'.

```

I am enrolling the fleet agent using the command without insecure I am not able setup the agent

```auto
sudo ./elastic-agent install --url=https://kibana.domain.app:8220 --enrollment-token=token==

```

Getting the below error

```auto
root@imaging:~/elastic-agent-8.13.4-linux-x86_64# sudo ./elastic-agent install --url=https://kibana.domain.app:8220 --enrollment-token=token==
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[=] Service Started [11s] Elastic Agent successfully installed, starting enrollment.
[===] Waiting For Enroll... [12s] {"log.level":"info","@timestamp":"2024-07-26T12:21:38.774Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://kibana.domain.app:8220/","ecs.version":"1.6.0"}
[==] Waiting For Enroll... [13s] {"log.level":"info","@timestamp":"2024-07-26T12:21:39.294Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":528},"message":"1st enrollment attempt failed, retrying for 10m0s, every 1m0s enrolling to URL: https://kibana.domain.app:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html
[==] Uninstalled [14s] Error uninstalling. Printing logs
.
.
{debug 2024-07-26 12:21:39.729530731 +0000 UTC m=+15.751428623 Loaded configuration from /root/elastic-agent-8.13.4-linux-x86_64/elastic-agent.yml github.com/elastic/elastic-agent/internal/pkg/config/loader.go:45 }
{debug 2024-07-26 12:21:39.729551801 +0000 UTC m=+15.751449692 Merged configuration from /root/elastic-agent-8.13.4-linux-x86_64/elastic-agent.yml into result github.com/elastic/elastic-agent/internal/pkg/config/loader.go:57 }
{debug 2024-07-26 12:21:39.729556839 +0000 UTC m=+15.751454731 Merged all configuration files from [/root/elastic-agent-8.13.4-linux-x86_64/elastic-agent.yml], no external input files github.com/elastic/elastic-agent/internal/pkg/config/loader.go:64 }
{debug 2024-07-26 12:21:39.729698729 +0000 UTC m=+15.751596620 composable Starting controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:118 }
{debug 2024-07-26 12:21:39.729714277 +0000 UTC m=+15.751612169 composable Started controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:161 }
{debug 2024-07-26 12:21:39.729805537 +0000 UTC m=+15.751703429 composable Kubernetes leaderelection provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetesleaderelection/kubernetes_leaderelection.go:53 }
{debug 2024-07-26 12:21:39.72981408 +0000 UTC m=+15.751711976 composable kubernetes_secrets provider skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetessecrets/kubernetes_secrets.go:81 }
{debug 2024-07-26 12:21:39.729831154 +0000 UTC m=+15.751729051 composable Variable state changed for composable inputs; debounce started github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:197 }
{debug 2024-07-26 12:21:39.729861325 +0000 UTC m=+15.751759218 composable.providers.kubernetes Kubernetes provider for resource pod skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes/kubernetes.go:106 }
{debug 2024-07-26 12:21:39.729857153 +0000 UTC m=+15.751755050 docker Docker client will negotiate the API version on the first request. github.com/elastic/elastic-agent-autodiscover@v0.6.7/docker/client.go:49 }
{debug 2024-07-26 12:21:39.729896172 +0000 UTC m=+15.751794065 composable.providers.kubernetes Kubernetes provider for resource node skipped, unable to connect: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable github.com/elastic/elastic-agent/internal/pkg/composable/providers/kubernetes/kubernetes.go:106 }
{info 2024-07-26 12:21:39.730076524 +0000 UTC m=+15.751974416 composable.providers.docker Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? github.com/elastic/elastic-agent/internal/pkg/composable/providers/docker/docker.go:44 }
{debug 2024-07-26 12:21:39.830392957 +0000 UTC m=+15.852290922 composable Computing new variable state for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:213 }
{debug 2024-07-26 12:21:39.830548547 +0000 UTC m=+15.852446495 composable Stopping controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:165 }
{debug 2024-07-26 12:21:39.930361552 +0000 UTC m=+15.952259497 composable Stopped controller for composable inputs github.com/elastic/elastic-agent/internal/pkg/composable/controller.go:194 }
Error: enroll command failed for unknown reason: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html

```

If I try to enroll with --insecure I am able to install the agent

```auto
root@coolr-imaging:~/elastic-agent-8.13.4-linux-x86_64# sudo ./elastic-agent install --url=https://kibana.domain.app:8220 --enrollment-token=token-here --insecure
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[==] Service Started [5s] Elastic Agent successfully installed, starting enrollment.
[===] Waiting For Enroll... [5s] {"log.level":"warn","@timestamp":"2024-07-26T12:44:25.401Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[==] Waiting For Enroll... [5s] {"log.level":"info","@timestamp":"2024-07-26T12:44:26.183Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://kibana.stream4tech.app:8220/","ecs.version":"1.6.0"}
[====] Waiting For Enroll... [6s] {"log.level":"warn","@timestamp":"2024-07-26T12:44:26.396Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[==] Waiting For Enroll... [7s] {"log.level":"info","@timestamp":"2024-07-26T12:44:27.405Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-07-26T12:44:27.407Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":300},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=] Done [7s]
Elastic Agent has been successfully installed.

```

Agent is showing healthy but there no metrics

This the status showing on kibana dashboard Below are the machines that enrolled inside proxmox server using private IP

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a040781a17ebf0654685222ea7ce234b48adb0f2.png)

There is a thing like if I enroll a machine in fleet server internally and externally I am able to enroll

```auto
root@sftp-server:~/elastic-agent-8.13.4-linux-x86_64# sudo ./elastic-agent install --url=https://192.168.13.168:8220 --enrollment-token=token-here --insecure
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:
[====] Service Started [5s] Elastic Agent successfully installed, starting enrollment.
[===] Waiting For Enroll... [5s] {"log.level":"warn","@timestamp":"2024-07-26T12:33:43.476Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[=] Waiting For Enroll... [5s] {"log.level":"info","@timestamp":"2024-07-26T12:33:43.649Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":519},"message":"Starting enrollment to URL: https://192.168.13.168:8220/","ecs.version":"1.6.0"}
[=] Waiting For Enroll... [5s] {"log.level":"warn","@timestamp":"2024-07-26T12:33:43.868Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
[=] Waiting For Enroll... [6s] {"log.level":"info","@timestamp":"2024-07-26T12:33:44.641Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
[=] Waiting For Enroll... [6s] {"log.level":"info","@timestamp":"2024-07-26T12:33:44.688Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":300},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=] Done [6s]                               
Elastic Agent has been successfully installed.
root@sftp-server:~/elastic-agent-8.13.4-linux-x86_64# 

```

Agent Policy under I am assigning new agents

```auto
Fleet Server Policy - ROHAN
rev. 14

```

Below is the logs that I am getting from elastic agent

**ISSUE - I am not able to get metrics from the machines that enrolled outside the network (proxmox) what is the certificate verificate part I am missing? and How can I fix**