Fleet server does not start properly

Servando_Dominguez · August 24, 2021, 3:56pm

Hi there! I'm running an ELK stack on prem version 7.14 with all the services in diferent VMs, running properly from 2 months ago and shipping the logs with filebeat, etc. Kibana/logstash uses production certs for the https but elasticsearch cluster uses autogenerated certs.

I deployed a new server in order to deploy a fleet server and move my filebeat inputs to elastic-agents but I get an error when I try to do this op.

The fleet server logs:

{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-24T15:40:05.516Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-24T15:40:05.516Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-24T15:40:06.03Z","message":"received initial configuration starting Fleet Server"}
{"log.level":"info","service.name":"fleet-server","cfg":{"NumCounters":500000,"MaxCost":52428800,"ActionTTL":300000000000,"ApiKeyTTL":900000000000,"EnrollKeyTTL":60000000000,"ArtifactTTL":86400000000000,"ApiKeyJitter":300000000000},"@timestamp":"2021-08-24T15:40:06.03Z","message":"makeCache"}
{"log.level":"info","service.name":"fleet-server","status":"STARTING","@timestamp":"2021-08-24T15:40:06.032Z","message":"Starting"}
{"service.name":"fleet-server","log.level":"info","log.logger":"fleet-metrics.api","message":"Starting stats endpoint","@timestamp":"2021-08-24T15:40:06.033Z"}
{"service.name":"fleet-server","log.level":"info","log.logger":"fleet-metrics.api","message":"Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock)","@timestamp":"2021-08-24T15:40:06.033Z"}
{"log.level":"info","service.name":"fleet-server","name":"VoxelInfraLogs","uuid":"e2GiZmRxTreCCF2gg_dqww","vers":"7.14.0","@timestamp":"2021-08-24T15:40:06.063Z","message":"Cluster Info"}
{"log.level":"info","service.name":"fleet-server","opts":{"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8,"blockQueueSz":32,"apikeyMaxParallel":120},"@timestamp":"2021-08-24T15:40:06.064Z","message":"Run bulker with options"}
{"log.level":"info","service.name":"fleet-server","fleet_version":"7.14.0","elasticsearch_version":"7.14.0","@timestamp":"2021-08-24T15:40:06.065Z","message":"versions are compatible"}
{"log.level":"info","service.name":"fleet-server","name":"VoxelInfraLogs","uuid":"e2GiZmRxTreCCF2gg_dqww","vers":"7.14.0","@timestamp":"2021-08-24T15:40:06.074Z","message":"Cluster Info"}
{"log.level":"info","service.name":"fleet-server","index":".fleet-policies","ctx":"index monitor","@timestamp":"2021-08-24T15:40:06.074Z","message":"start"}
{"log.level":"info","service.name":"fleet-server","ctx":"policy agent monitor","throttle":5,"@timestamp":"2021-08-24T15:40:06.074Z","message":"run policy monitor"}
{"log.level":"info","service.name":"fleet-server","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-08-24T15:40:06.074Z","message":"start"}
{"log.level":"info","service.name":"fleet-server","limits":{"Interval":1000000,"Burst":1000,"Max":0,"MaxBody":1048576},"long_poll_timeout":300000,"long_poll_timestamp":30000,"long_poll_jitter":30000,"@timestamp":"2021-08-24T15:40:06.074Z","message":"Checkin install limits"}
{"log.level":"info","service.name":"fleet-server","limits":{"Interval":10000000,"Burst":100,"Max":50,"MaxBody":524288},"@timestamp":"2021-08-24T15:40:06.074Z","message":"Enroller install limits"}
{"log.level":"info","service.name":"fleet-server","limits":{"Interval":5000000,"Burst":25,"Max":50,"MaxBody":0},"maxParallel":8,"@timestamp":"2021-08-24T15:40:06.074Z","message":"Artifact install limits"}
{"log.level":"info","service.name":"fleet-server","limits":{"Interval":10000000,"Burst":100,"Max":50,"MaxBody":2097152},"@timestamp":"2021-08-24T15:40:06.074Z","message":"Ack install limits"}
{"log.level":"info","service.name":"fleet-server","method":"GET","path":"/api/status","@timestamp":"2021-08-24T15:40:06.074Z","message":"Server install route"}
{"log.level":"info","service.name":"fleet-server","method":"POST","path":"/api/fleet/agents/:id","@timestamp":"2021-08-24T15:40:06.074Z","message":"Server install route"}
{"log.level":"info","service.name":"fleet-server","method":"POST","path":"/api/fleet/agents/:id/checkin","@timestamp":"2021-08-24T15:40:06.074Z","message":"Server install route"}
{"log.level":"info","service.name":"fleet-server","method":"POST","path":"/api/fleet/agents/:id/acks","@timestamp":"2021-08-24T15:40:06.074Z","message":"Server install route"}
{"log.level":"info","service.name":"fleet-server","method":"GET","path":"/api/fleet/artifacts/:id/:sha2","@timestamp":"2021-08-24T15:40:06.074Z","message":"Server install route"}
{"log.level":"info","service.name":"fleet-server","bind":"0.0.0.0:8220","rdTimeout":60000,"wrTimeout":600000,"@timestamp":"2021-08-24T15:40:06.074Z","message":"server listening"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-24T15:40:06.075Z","message":"server hard connection limiter disabled"}
{"log.level":"warn","service.name":"fleet-server","ctx":"policy leader manager","@timestamp":"2021-08-24T15:40:06.075Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled"}
{"log.level":"error","service.name":"fleet-server","index":".fleet-policies","ctx":"index monitor","error.message":"elastic fail 500:exception:[service account role descriptor resolving] requires TLS for the HTTP interface","@timestamp":"2021-08-24T15:40:06.082Z","message":"failed to initialize the global checkpoints"}
{"log.level":"info","service.name":"fleet-server","index":".fleet-policies","ctx":"index monitor","error.message":"elastic fail 500:exception:[service account role descriptor resolving] requires TLS for the HTTP interface","@timestamp":"2021-08-24T15:40:06.082Z","message":"exited"}
{"log.level":"info","service.name":"fleet-server","ctx":"policy agent monitor","@timestamp":"2021-08-24T15:42:04.575Z","message":"Exit policy monitor local"}
{"log.level":"info","service.name":"fleet-server","status":"STOPPING","@timestamp":"2021-08-24T15:42:04.575Z","message":"Stopping"}
{"log.level":"error","service.name":"fleet-server","error.message":"elastic fail 500:exception:[service account role descriptor resolving] requires TLS for the HTTP interface","@timestamp":"2021-08-24T15:42:04.575Z","message":"Policy index monitor exited"}
{"log.level":"info","service.name":"fleet-server","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-08-24T15:42:04.575Z","message":"context closed waiting for global checkpoints advance"}
{"log.level":"info","service.name":"fleet-server","index":".fleet-actions","ctx":"index monitor","@timestamp":"2021-08-24T15:42:04.575Z","message":"exited"}
{"service.name":"fleet-server","log.level":"info","log.logger":"fleet-metrics.api","message":"Stats endpoint (/var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection","@timestamp":"2021-08-24T15:42:04.575Z"}
{"log.level":"info","service.name":"fleet-server","error.message":"elastic fail 500:exception:[service account role descriptor resolving] requires TLS for the HTTP interface","@timestamp":"2021-08-24T15:42:04.575Z","message":"Fleet Server exited"}
{"log.level":"info","service.name":"fleet-server","status":"STARTING","@timestamp":"2021-08-24T15:42:04.575Z","message":"Starting"}
{"log.level":"info","service.name":"fleet-server","status":"STOPPING","@timestamp":"2021-08-24T15:42:04.575Z","message":"Stopping"}
{"service.name":"fleet-server","log.level":"info","log.logger":"fleet-metrics.api","message":"Starting stats endpoint","@timestamp":"2021-08-24T15:42:04.575Z"}
{"service.name":"fleet-server","log.logger":"fleet-metrics.api","message":"Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock (configured: unix:///var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock)","log.level":"info","@timestamp":"2021-08-24T15:42:04.576Z"}
{"log.level":"info","service.name":"fleet-server","name":"VoxelInfraLogs","uuid":"e2GiZmRxTreCCF2gg_dqww","vers":"7.14.0","@timestamp":"2021-08-24T15:42:04.585Z","message":"Cluster Info"}
{"log.level":"error","service.name":"fleet-server","error.message":"context canceled","@timestamp":"2021-08-24T15:42:04.585Z","message":"failed to fetch elasticsearch version"}
{"log.level":"info","service.name":"fleet-server","opts":{"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8,"blockQueueSz":32,"apikeyMaxParallel":120},"@timestamp":"2021-08-24T15:42:04.585Z","message":"Run bulker with options"}
{"service.name":"fleet-server","message":"Stats endpoint (/var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock) finished: accept unix /var/lib/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock: use of closed network connection","log.level":"info","log.logger":"fleet-metrics.api","@timestamp":"2021-08-24T15:42:04.585Z"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-24T15:42:04.585Z","message":"Fleet Server exited"}

I execute the deploy of the fleet server with this command:
sudo elastic-agent enroll -d debug -f --fleet-server-es=https://<elastic_ip>:9200 --fleet-server-service-token=<proper_token> --fleet-server-policy=id:5fa328c0-f635-11eb-b62c-2d3279f943d3 --fleet-server-es-ca=/etc/elastic-agent/certs/elasticsearch-ca.pem

Here the elastic-agent logs:

[...]
{"log.level":"info","@timestamp":"2021-08-24T15:40:05.115Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is rEMABmG_","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:40:05.115Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:40:05.158Z","log.origin":{"file.name":"operation/operator.go","file.line":259},"message":"operation 'operation-install' skipped for fleet-server.7.14.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:40:05.493Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-08-24T17:40:05+02:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:40:05.494Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:42:04.574Z","log.origin":{"file.name":"cmd/run.go","file.line":189},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:42:04.574Z","log.origin":{"file.name":"operation/operator.go","file.line":191},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:42:04.574Z","log.origin":{"file.name":"process/app.go","file.line":181},"message":"Signaling application to stop because of shutdown: fleet-server--7.14.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:42:05.075Z","log.origin":{"file.name":"cmd/run.go","file.line":197},"message":"Shutting down completed.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:42:05.075Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-08-24T17:42:05+02:00 - message: Application: fleet-server--7.14.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-08-24T15:42:05.075Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":66},"message":"Stats endpoint (/var/lib/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /var/lib/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection","ecs.version":"1.6.0"}

Could anyone give me some light?

Thanks in advance,

V

ruflin · August 25, 2021, 7:38am

I'm stumbling over the following log line:

{"log.level":"info","service.name":"fleet-server","index":".fleet-policies","ctx":"index monitor","error.message":"elastic fail 500:exception:[service account role descriptor resolving] requires TLS for the HTTP interface","@timestamp":"2021-08-24T15:40:06.082Z","message":"exited"}

But I also see you are connecting through https to Elasticsearch is this correct? What is your TLS config in Elasticsearch?

blaker · August 25, 2021, 1:39pm

@ruflin Could be possible that the HTTPS is managed by a HTTP reverse proxy and not by Elasticsearch and that is why that error is showing. Just a thought.

Servando_Dominguez · August 26, 2021, 12:59pm

Hi all,

@blaker There is no reverse proxy in that installation.

@ruflin I use the autogenerated certificates way. Here the configuration with the autogenerated certs of the master node:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/AVVLS-ELAST-001.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-stack-ca.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/certs/http.p12"

The installation is in a rocky linux. I tried a clean installation with the same installation procedure and works properly. Maybe is an update issue.

Servando_Dominguez · August 29, 2021, 9:07pm

I did a new installation and migrate the data. We can close this case. The workaround was hard.

ruflin · August 30, 2021, 7:17am

Sorry for the bumpy road @Servando_Dominguez The reinstall with all the same settings just worked?

system · September 27, 2021, 7:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fleet server install failed on fresh ELK stack 8.2.0 Elasticsearch fleet , elastic-agent	5	381	June 8, 2022
Fleet for collecting log Elastic Observability elastic-stack-monitoring , docker	2	335	March 25, 2024
Fleet Server Fails At Start, No Reason Given Beats fleet	2	759	April 8, 2022
Attempting to start Fleet Server fails Beats fleet	6	2240	October 24, 2021
Issues with Fleet Server (Elastic Agent Setup) in local Elasticsearch deployment with self signed certificates Beats docker , fleet , elastic-agent	2	1361	July 16, 2021

Fleet server does not start properly

Related Topics