Unable to Install Fleet Server - EOF

Elastic Stack Beats
1

Hello, I'm hitting a wall trying to install Fleet Server on the same host as my ELK stack (v8.5.2).

The result is always the same:

Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:
{"log.level":"info","@timestamp":"2022-11-25T18:14:05.343Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-25T18:14:09.344Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - EOF","ecs.version":"1.6.0"}

When enrolling with:

sudo ./elastic-agent install  --fleet-server-es=http://localhost:9200 \
  --fleet-server-service-token=<token> \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=<sha256fingerprint>

This same error also occurs when using the flags:
--url=https://10.0.1.25:8220 --fleet-server-es-ca=/opt/elk/secrets/certs/ca/ca.crt --certificate-authorities=/opt/elk-fs/secrets/certs/ca/ca.crt --fleet-server-cert=/opt/elk-fs/secrets/certs/fleet-server/fleet-server.crt --fleet-server-cert-key=/opt/elk-fs/secrets/certs/fleet-server/fleet-server.key
and
--fleet-server-es-insecure or --insecure

All these flags having been suggested in various support channels related to this issue.

What else might I be able to do, to troubleshoot?

Thank you.

2

Settings in Kibana:

CleanShot 2022-11-25 at 13.31.35
CleanShot 2022-11-25 at 13.31.351346×896 95.5 KB

3

Hi @tmeuze Welcome to the community! ... Apologies you are having trouble.

What Version?

So you can not install fleet without security enabled, did you enable security by default?

If so then Elasticsearach should be running on https not http

I see

--fleet-server-es=http://localhost:9200

should be https

--fleet-server-es=https://localhost:9200

Also you will probably want to bind elasticsearch to the network so agents on other host and then you and ip address or host name in that settings otherwise no remote agent will be able to connect. If You did that (default installation) that should have already been set..

in elasticsearch.yml
http.host: 0.0.0.0

Try to use the defaults....

Get it working then adjust... I think all those defaults should have been setup if you did a default install of Elasticsearch and Kibana with security enabled

4

Here is a nice long thread on debugging this from just yesterday... took us many steps to find that simple mistake I would recommend reading through this..

1 Like
5

Hello and happy Thanksgiving (if you celebrate)! Thanks for the quick response.

Also, thank you for pointing out the error. I had to update the Outputs host to reflect https://localhost:9200 in order to correct that. Now the fleet server install wizard prints https.

I successfully installed using

sudo ./elastic-agent install \
  --url=https://10.0.1.25:8220 \
  --fleet-server-es=https://localhost:9200 \
  --fleet-server-service-token=<token> \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca=/opt/elk/secrets/certs/ca/ca.crt \
  --certificate-authorities=/opt/elk-fs/secrets/certs/ca/ca.crt \
  --fleet-server-cert=/opt/elk-fs/secrets/certs/fleet-server/fleet-server.crt \
  --fleet-server-cert-key=/opt/elk-fs/secrets/certs/fleet-server/fleet-server.key

And adding ssl.certificate_authorities: ["/opt/elk/secrets/certs/ca/ca.crt"] to my Output advanced YAML config.

Adding agents using:

sudo ./elastic-agent install --url=https://10.0.1.25:8220 --enrollment-token=<token> --certificate-authorities=/opt/elk/secrets/certs/ca/ca.crt

AFTER having placed the Elasticsearch CA in the same exact location as specified for the Fleet Server.

Now I have both a healthy FS and Agent. Logs are streaming in for the Fleet Server, but not for the agent.

I have checked the ES logs to make sure I wasn't getting a "bad certificate" error (as it had been an issue previously), but I don't see any such complaints now. elastic-agent status on the agent-side shows all four applications as healthy.

Any advice on how I might troubleshoot from here?

Thank you.

EDIT: This is version 8.5.2 for the ELK stack and for the agents.

6

Good progress and thank you!

Per that other thread run the elastic-agent collect diagnostics where the agent is running and look and look at the troubleshooting docs I referenced.

Unzips the diagnostics and look through the logs and configs.

If the agent is on a remote host from elasticsearch the default output localhost won't work..

That was exact problem. Getting yesterday's threads may not be yours, but you should be able to dig into those diagnostics and figure it out again. It's something simple at this point

I gave tips for debugging in that other thread.

7

I did make the same mistake as in the other thread. I changed the Output to the fixed IP address to resolve this. However - I think my progress is diverging from the other thread. I am now getting a bad_certificate java error as described here.

This happens with or without the --insecure flag.

I originally specified the CA certificate file for Fleet Server to use (under Advanced YAML Confiugration), which I generated separately from the native ES CA. I also tried specifying the SHA256 fingerprint of the generated CA, then passing the --fleet-server-es-ca-trusted-fingerprint for agent install. This flag fails without also using --insecure (due to unknown CA).

It seems to me that ES does not trust this CA that I generated, using the elasticsearch-certutil and these docs. Am I on the right track? These docs appear not to mention how to make ES trust the CA it instructs me to generate.

Full transparency:
I am not using the native ES CA because I am running ELK using a special docker setup, and can't quite figure out how to retrieve the CA key password.

I will continue researching how to resolve this.

Thank you for your help and time!

8

Well Then... glad we got that out in the open :slight_smile:

Generally when I see I am using a special / custom setup then you assume all the responsibility to understand it and you will need to dig in and see what is going on... and understand it.... in detail....

Yes you are on the right track but you did not actually look at the code that generates the CA and the certs

It generates a random password for the CA ... uses it as needed and then poof gone into the ether... you will never decode the CA or be able to use it to generate more certs.

The good thing is the scripts is using PEM format not .p12 ...

You could take out the password and then use the CA to generate other certs

You could save / set the password and then use the CA to generate other certs

You could echo out the password and then use the CA to generate other certs

In general mixing CA is OK if you are 3rd Level Wizard with Certs I am only 2nd Level ...

The way you would possibly mix the certs is you will need to list all the CAs where CAs are asked for ... now you are editing the elasticsearch.yml (which is fine if you are good with that).

I think if I were you I would echo out that password and the try to see if you can then use the CA to generate the other Certs.

BUT then ... when I look a bit closer (I am not sure you even need to do that)
On the page you referenced

It appears to say you should just be able to use the generated es CA generated from the docker

So did you just try to set the

ssl.certificate_authorities: ["/path/to/your/elasticsearch-ca.crt"]

as instructed

elasticsearch-ca.crt The CA certificate to use to connect to Elasticsearch. This is the CA used to generate certs for Elasticsearch (see Prerequisites).

Note that the CA certificate’s SHA-256 fingerprint (hash) may be used instead of the elasticsearch-ca.crt file for securing connections to Elasticsearch.

I

Otherwise, under Advanced YAML configuration, set ssl.certificate_authorities and specify the CA certificate to use to connect to Elasticsearch. You can specify a list of file paths (if the files are available), or embed a certificate directly in the YAML configuration. If you specify file paths, the certificates must be available on the hosts running the Elastic Agents.

File path example:

ssl.certificate_authorities: ["/path/to/your/elasticsearch-ca.crt"]

The path to the CA certificate on the Elastic Agent host.

Which you should be able to get from the docker setup.

And in the end... you will Elasticsearch and Kibana in the Special Docker.. and Fleet standalone... hmm

But really I am a bit guessing here as I do not have a replica of the setup...

Are you using the other stuff in the "Special Docker"???

I noticed some of the Nice to Haves / Fixes are already standard in new Kibana etc...

9

First off - thank you again so very much for your help. You definitely went above-and-beyond in helping me, when you could've said "kick rocks - this is not supported".

You are a genius! Echoing the password worked. Moreover, because this "special" setup uses the official docker stack, I was able to leverage instances.yml to set up the Fleet Server certificate - signed by the ES CA - at runtime.

Everything is working properly now, and I think your helping me out here might in turn help others out - if I can only walk back through my steps again.

Thank you again, Stephen. I hope you have a great rest of your holidays.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.