Hello,
After spending hours trying different types of configurations and running in circles through all the documentation I will try to find help here.
The following setup is done completely on Proxmox. It should help us to prepare for deployment of the Elasticsearch into production.
I have cluster of 3 nodes of Elasticsearch:
elk-1 - IP 10.212.25.197
elk-2 - IP 10.212.25.198
elk-3 - IP 10.212.25.199
I have one Kibana node:
kibana - IP 10.212.25.200
The traffic between the cluster and Kibana is secured by the http_ca.crt certificate autogenerated during the installation of first cluster node elk-1. I am able to login to Kibana and I can see all 3 nodes of cluster under Management > Stack Monitoring. All is green and I do monitor the cluster nodes with Metricbeat. I can also see the Kibana, which is self monitored.
I would like to use the Integration > Linux Metrics to monitor the RedHat server. For this I need to configure the fleet server. I want the fleet server to be on separated device, so I am using another VM named:
logstash - IP 10.212.25.201
1, In Kibana under Fleet > Agents > I did choose "Advanced".
2, Then "Production" deployment mode for security.
3, I did add Fleet server host "https://10.212.25.201:8220"
4, I did generate Service token and did get the following config for agent installation.
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.1-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.4.1-linux-x86_64.tar.gz
cd elastic-agent-8.4.1-linux-x86_64
sudo ./elastic-agent install --url=https://10.212.25.201:8220 \
--fleet-server-es=https://10.212.25.197:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM2MDY2Nzg2Mzk6WHN4Qm1tVHRSNW04bEJUemZrWWtwQQ \
--fleet-server-policy=fleet-server-policy \
--certificate-authorities=<PATH_TO_CA> \
--fleet-server-es-ca=<PATH_TO_ES_CERT> \
--fleet-server-cert=<PATH_TO_FLEET_SERVER_CERT> \
--fleet-server-cert-key=<PATH_TO_FLEET_SERVER_CERT_KEY>
- Next I did generate a new certificate authority on elk-1 cluster node
/usr/share/elasticsearch/bin/elasticsearch-certutil ca --pem
- Used the certificate authority to generate certificates for Fleet Server on elk-1 cluster node
/usr/share/elasticsearch/bin/elasticsearch-certutil cert\
--name fleet-server \
--ca-cert /home/ttyser/ca/ca.crt \
--ca-key /home/ttyser/ca/ca.key \
--dns logstash \
--ip 10.212.25.201 \
--pem
- Did copy the certificate-bundle.zip, ca.crt, http_ca.crt from node elk-1 to fleetserver. On fleetserver created under /home/ttyser/ directory "certs" and moved all the certs inside and unziped the bundle.zip. So it looks like this:
root@logstash:/home/ttyser/certs# ls
ca.crt certificate-bundle.zip fleet-server http_ca.crt
root@logstash:/home/ttyser/certs# ls fleet-server/
elastic-agent-8.4.1-linux-x86_64 elastic-agent-8.4.1-linux-x86_64.tar.gz fleet-server.crt fleet-server.key
- I did customize the config for agent:
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.1-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.4.1-linux-x86_64.tar.gz
cd elastic-agent-8.4.1-linux-x86_64
sudo ./elastic-agent install --url=https://10.212.25.201:8220 \
--fleet-server-es=https://10.212.25.197:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM2MDY2Nzg2Mzk6WHN4Qm1tVHRSNW04bEJUemZrWWtwQQ \
--fleet-server-policy=fleet-server-policy \
--certificate-authorities=/home/ttyser/certs/ca.crt \
--fleet-server-es-ca=/home/ttyser/certs/http_ca.crt \
--fleet-server-cert=/home/ttyser/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/home/ttyser/certs/fleet-server/fleet-server.key
- I tried to install the Fleetserver:
root@logstash:/home/ttyser/certs/fleet-server# curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.4.1-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.4.1-linux-x86_64.tar.gz
cd elastic-agent-8.4.1-linux-x86_64
sudo ./elastic-agent install --url=https://10.212.25.201:8220 \
--fleet-server-es=https://10.212.25.197:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM2MDY2Nzg2Mzk6WHN4Qm1tVHRSNW04bEJUemZrWWtwQQ \
--fleet-server-policy=fleet-server-policy \
--certificate-authorities=/home/ttyser/certs/ca.crt \
--fleet-server-es-ca=/home/ttyser/certs/http_ca.crt \
--fleet-server-cert=/home/ttyser/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/home/ttyser/certs/fleet-server/fleet-server.key
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 382M 100 382M 0 0 15.3M 0 0:00:24 0:00:24 --:--:-- 15.8M
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/elastic-agent
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/endpoint-security-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/fleet-server-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/elastic-agent.yml
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/cloudbeat-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/filebeat-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/.elastic-agent.active.commit
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/metricbeat-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/cloudbeat-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/endpoint-security-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/fleet-server-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/metricbeat-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/.build_hash.txt
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/fleet-server-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/osquerybeat-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/metricbeat-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/apm-server-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/cloudbeat-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/apm-server-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/filebeat-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/heartbeat-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/endpoint-security-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/README.md
elastic-agent-8.4.1-linux-x86_64/elastic-agent.reference.yml
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/osquerybeat-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/osquerybeat-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/NOTICE.txt
elastic-agent-8.4.1-linux-x86_64/LICENSE.txt
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/heartbeat-8.4.1-linux-x86_64.tar.gz.sha512
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/apm-server-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/heartbeat-8.4.1-linux-x86_64.tar.gz.asc
elastic-agent-8.4.1-linux-x86_64/data/elastic-agent-8d7885/downloads/filebeat-8.4.1-linux-x86_64.tar.gz
elastic-agent-8.4.1-linux-x86_64/elastic-agent
Installed as a system package, installation will not be altered.
{"log.level":"info","@timestamp":"2022-09-19T17:36:22.019Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on active enrollment keys to be created in policy with Fleet Server integration: fleet-server-policy","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T17:36:24.021Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":759},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T17:36:28.030Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on active enrollment keys to be created in policy with Fleet Server integration: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed for unknown reason: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
- Can someone tell me what is the correct setup to do this?
Today I tried again. Created brand new VM called fleetserver, created brand new certificates for fleet server and add elasticsearch fingerprint in Kibana fleetserver. Here is the result:
root@fleetserver:~/elastic-agent-8.4.1-linux-x86_64# sudo ./elastic-agent install --url=https://10.212.25.201:8220 \
--fleet-server-es=https://10.212.25.197:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM2NTg0MzcyMTU6UzZwQktEWFZTcnE1eXRJa1BwcGx2UQ \
--fleet-server-policy=fleet-server-policy \
--fleet-server-es-ca-trusted-fingerprint=e32b2c48128a3fe08b3490562cf633a7b2745fbc553ef3f3e164f6f3a9ea2040 \
--certificate-authorities=/home/ttyser/certs/ca.crt \
--fleet-server-cert=/home/ttyser/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/home/ttyser/certs/fleet-server/fleet-server.key
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-09-20T07:25:01.073Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":759},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-20T07:25:07.095Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
At this point not really sure what more to do/try. I believe someone here had to do this in the past and he has the process in his head. It would be super awesome if someone could share it.
UPDATE: Found some discussion about not using auto generated policy and used manually generated on and add fleetserver as integration. Did that, now the policy in generated config looks better I would say. The output of the installation process changed, but still finished with error.
root@fleetserver:/etc/elastic-agent-8.4.1-linux-x86_64# sudo ./elastic-agent install --url=https://10.212.25.201:8220 \
--fleet-server-es=https://10.212.25.197:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM2NjU2NzM2NjM6bEplUUNTQkVTbUM0azh1em9OY0FxZw \
--fleet-server-policy=0bc55d30-38c5-11ed-b16d-936192d8efc7 \
--fleet-server-es-ca-trusted-fingerprint=e32b2c48128a3fe08b3490562cf633a7b2745fbc553ef3f3e164f6f3a9ea2040 \
--certificate-authorities=/home/ttyser/certs/ca.crt \
--fleet-server-cert=/home/ttyser/certs/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/home/ttyser/certs/fleet-server/fleet-server.key
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-09-20T06:47:27.455Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":447},"message":"Retrying to restart...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-20T06:47:28.468Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":773},"message":"Fleet Server - Running on policy with Fleet Server integration: 0bc55d30-38c5-11ed-b16d-936192d8efc7; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-20T06:47:29.236Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://10.212.25.201:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp 10.212.25.201:8220: connect: connection refused
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
root@fleetserver:/etc/elastic-agent-8.4.1-linux-x86_64#
Could this problem have something to do with the fact that when generating the certificates for fleetserver I did used hostname as dns? Should I use IP 10.212.25.201 or url https://10.212.25.201:8220?
/usr/share/elasticsearch/bin/elasticsearch-certutil cert\
--name fleet-server \
--ca-cert /home/ttyser/ca/ca.crt \
--ca-key /home/ttyser/ca/ca.key \
--dns fleetserver \
--ip 10.212.25.201 \
--pem
FIXED: It was the problem with the DNS in certificates. I did add following entries into /etc/hosts file on all my VMs.
10.212.25.197 elk-1.testing.lab
10.212.25.198 elk-2.testing.lab
10.212.25.199 elk-3.testing.lab
10.212.25.200 kibana.testing.lab
10.212.25.201 fleetserver.testing.lab
Then did recreate the certificates for the fleetserver:
/usr/share/elasticsearch/bin/elasticsearch-certutil cert\
--name fleet-server \
--ca-cert /home/ttyser/ca/ca.crt \
--ca-key /home/ttyser/ca/ca.key \
--dns fleetserver.testing.lab \
--ip 10.212.25.201 \
--pem
After that I did copy the new bundle certificate to fleetserver 10.212.25.201, unziped it into proper directories. Then went into Fleet policy i did create in Kibana web GUI and clicked Actions > Add agent and generated new installation config. I did customized new configuration with correct path to new certificates and finally the installation went through.
root@fleetserver:/etc/elastic-agent-8.4.1-linux-x86_64# sudo ./elastic-agent install --url=https://10.212.25.201:8220 \
--fleet-server-es=https://10.212.25.197:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM2NzIwNzY5OTY6WWhnblVFM0JRYU9feTAxdEFpRUVuUQ \
--fleet-server-policy=0bc55d30-38c5-11ed-b16d-936192d8efc7 \
--fleet-server-es-ca-trusted-fingerprint=e32b2c48128a3fe08b3490562cf633a7b2745fbc553ef3f3e164f6f3a9ea2040 \
--certificate-authorities=/home/ttyser/certs2/ca.crt \
--fleet-server-cert=/home/ttyser/certs2/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/home/ttyser/certs2/fleet-server/fleet-server.key
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-09-20T11:10:15.930Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":759},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-20T11:10:21.944Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":773},"message":"Fleet Server - Running on policy with Fleet Server integration: 0bc55d30-38c5-11ed-b16d-936192d8efc7; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-20T11:10:22.543Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://10.212.25.201:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-20T11:10:24.663Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":273},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.
It was terrible experience, but I learned a lot. Hope this will help someone, because I spent around 20 hours on it 