Elastic Agent - problems following the documentation, stumbled upon dead links and more

In the documentation for setting up Elastic Agent the following is stated:

Kibana user with All privileges on Fleet and Integrations. Since many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.

But when I select the user kibana, it shows that kibana is deprecated - that's fine, I check kibana_system, but I can't edit anything with it because it's a built-in user... I tried ignoring that thinking it's outdated and carried on... (please note that I did that AFTER I did the paragraph below)

I installed the Elastic Agent and started it with systemctl start elastic-agent and enabled it on boot with systemctl start elastic-agent.

But then I noticed the included documentation of Elastic Agent with a dead link... I tried stopping and disabling the Elastic Agent and I succeeded after a surprisinging big time leap. Then I followed the included README.md and ran

./elastic-agent -c elastic-agent.yml -e

which gave me the following output:

Error: could not read configuration file /var/lib/elastic-agent/elastic-agent.yml: open /var/lib/elastic-agent/elastic-agent.yml: no such file or directory
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.5/fleet-troubleshooting.html

Next I'm supposed to run this:
./elastic-agent setup -e

...but it doesn't make sense, since the config for Elastic Agent is clearly missing.

  • How do I solve these issues?..

My goal is to setup Elastic Agent with sample data for simple SIEM analysis (I'm still a beginner).

Hi @SirMuffington

Can you show us the document you're referring to?

Are you trying to install elastic agent as standalone or fleet managed?

The file is located in /usr/share/elastic-agent/README.md from the newest Elastic Agent registration. It was installed from a DEB package (running on Debian 11). The file's contents are:

# Welcome to Elastic-Agent 8.5.1

Agent manages other beats based on configuration provided.

## Getting Started

To get started with Elastic-Agent, you need to set up Elasticsearch on
your localhost first. After that, start Elastic-Agent with:

     ./elastic-agent -c elastic-agent.yml -e

This will start Elastic-Agent and send the data to your Elasticsearch
instance. To load the dashboards for Elastic-Agent into Kibana, run:

    ./elastic-agent setup -e

For further steps visit the
[Quick start](https://www.elastic.co/guide/en/beats/elastic-agent/main/elastic-agent-installation-configuration.html) guide.

## Documentation

Visit [Elastic.co Docs](https://www.elastic.co/guide/en/beats/elastic-agent/main/index.html)
for the full Elastic-Agent documentation.

## Release notes

https://www.elastic.co/guide/en/beats/libbeat/main/release-notes-8.5.1.html

I found the config in /etc/elastic-agent/elastic-agent.yml though. I guess that's my ticket to success?

I am trying to install it as fleet managed.

I would not use that Read Me (It looks to be more of a leftover from Beats)

That is not referring to Fleet managed

I would use the official documentation that I linked you to... That is why I provided it.

There's very easy steps to get fleet managed agent.

In fact, if you log into Kibana into Fleet, there are step by step instructions.

So I don't even need to grant these rights as told in the docs everywhere (including the link you sent in here)?:

A Kibana user with All privileges on Fleet and Integrations. Since many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.

I am just saying follow the documentation on our site here not the readme file.

With respect to the Role Based Control... you can start out using the default elastic user which has that role /privilege and it should all work or you can set up your own users and roles following the documentation that is up to you and then yes the user would need a role as described.

Alright, thank you for this information.

elastic works fine for now.

The problem I am now facing is that Installed as a system package, installation will not be altered. gets printed out when I try to run all of the steps in the setup of the Fleet Server, which apparently needs to be setup by me. I did apt-get remove elastic-agent because I used a DEB package beforehand and then executed the elastic-agent from the tar and then this got printed out and it got stuck:

{"log.level":"info","@timestamp":"2022-11-22T21:50:28.912Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T21:50:31.313Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-22T21:50:35.315Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - EOF","ecs.version":"1.6.0"}

Why is it indicating an end of file?

Hi @SirMuffington

Apologies I am not following I think we need to back up a bit I may have made an assumption... I did not realize you are actually trying to install the Fleet Server itself... which is a particular policy / type of elastic agent ... I thought you were just trying to install the normal agent to collect / logs metrics etc. after you already. had a fleet server...

Installing the Fleet server itself has a specific set of install instructions.

So a couple questions to confirm ...

What does that mean.....

Do you have a self managed Elasticsearch and Kibana Running with Authentication and TLS enabled and the version is 8.5.1? i.e. all the prerequisites? here

And next you want to install a Fleet Server (elastic-agent with the fleet server policy? Is that what you are trying to accomplish?

I just followed the in the Docs below and in Kibana and installed and connected the fleet server immediately took about 5 mins...

If you want to install your own fleet server you need to follow the instructions here

Are those the instructions you followed? Did you use the Quickstart or Advanced ... notice that it show installing with the tar.gz and if you follow the steps it should give the correct commands to run.

So to help I will need to know what you have exactly what steps you followed.

Here is what I did .. I just copied the commands from Kibana then ran them and it installed and connected. It will not like it if you already have an agent running you will want to uninstall that and install with this command (you can then later add other policies to this agent)

Note this is on an Elasticsarch / Kibana 8.5.2 on Ubuntu 20.04 , Default Setup with Self Signed Certs

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.5.2-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.5.2-linux-x86_64.tar.gz
cd elastic-agent-8.5.2-linux-x86_64
sudo ./elastic-agent install \
  --fleet-server-es=https://10.168.0.12:9200 \
  --fleet-server-service-token=AdsfgdsfgdsfgdsfgdsfgZXQtc2VydmVyL3Rva2VuLTE2NjkxNjM2NDk0Njg6Tks2bmpwN0ZTWmk5N3dBYm04eG0wZw \
  --fleet-server-policy=fleet-server-policy \
  --fleet-server-es-ca-trusted-fingerprint=sdfgdsfgdsfgsdfgs57ba07ea89756df453c42d1659442c1d7f192d
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
{"log.level":"info","@timestamp":"2022-11-23T00:37:04.252Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":403},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-23T00:37:06.792Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-23T00:37:10.795Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":773},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-23T00:37:11.204Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://stephenb-es-8-test:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-11-23T00:37:12.559Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":273},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

I did not get any messages about generating self signed cert...

So let me know the exact steps you are taking, show the exact commands and what is not working... and perhaps me or someone else can help

Thank you for your kind reply.

By that I meant that Kibana with Elastic on self-signed certificates and apparently over HTTP (even though the certificates exist) work. I don't care about SSL right now, since this is only a testing setup. My main goal is to understand how Elastic is being used for SIEM and not all the SSL stuff. It's important too, but first things first..

I think we're on the same page now.

The current problem I'm facing is in the reply above with the three lines of log entries in which the last one is an EOF error... Any idea how to fix this?

I retried this recently and it still spits out the same error...

BTW we found that if you use http in the above setting it results in that EOF error you can look at this pretty detailed thread about debugging this here and another detailed thread here

you will also need to add the flag to the install command.
--insecure to the install command if you do not use the fingerprint.

So even though SSL is not important to you at this time SSL is a requirement for Fleet / Agent etc it will not work over http, SSL is required.

I do see some reference to not needed TLS Transport for Quick Start but I am not familiar with that.... checking on that

In the end in 8.x Elastic has changed and it is "Secure By Default" and that is required for Fleet and Agent.

If you really just want to try all this is the least amount of you could just set up and Elastic Cloud Trial and all this would be done.. then you would just install the agents you want to collect the telemetry

@SirMuffington

ACTUALLY I stand Corrected!!! :slight_smile:
(I am always making sure everything is secure)

If you only have HTTP then you can connect not recommended but you can..

Look on this page Elastic Agent command reference | Fleet and Elastic Agent Guide [8.5] | Elastic

look at the insecure settings... I have never used them but perhaps that will work for you

--fleet-server-es-insecure

Allows fleet server to connect to Elasticsearch in the following situations:

  • When connecting to an HTTP server.
  • When connecting to an HTTPs server and the certificate chain cannot be verified. The content is encrypted, but the certificate is not verified.

When this flag is used the certificate verification is disabled.

--fleet-server-host <string>
Fleet Server HTTP binding host (overrides the policy).
--fleet-server-insecure-http
Expose Fleet Server over HTTP. This option is not recommended because it’s insecure. It’s useful during development and testing, but should not be used in production. When using this option, you should bind Fleet Server to the local host (this is the default).

I have used this with other agents on the remote hosts when needed.

--insecure

Allow the Elastic Agent to connect to Fleet Server over insecure connections. This setting is required in the following situations:

  • When connecting to an HTTP server. The API keys are sent in clear text.
  • When connecting to an HTTPs server and the certificate chain cannot be verified. The content is encrypted, but the certificate is not verified.
  • When using self-signed certificates generated by Elastic Agent.

We strongly recommend that you use a secure connection.

so give them a try... I won't be able to help much there... but perhaps that will get you over the hump.

Tried adding --install with no good results.

When I do now it says:
Error: already installed at: /opt/Elastic/Agent

And it doesn't go past step 2.

Not --install

And yes you will need to uninstall first and then reinstall.

Sorry, I mistyped, I meant --insecure which I tried again and it doesn't work...

I even tried adding both of the options together like this:

sudo ./elastic-agent install \
  --insecure \
  --fleet-server-es-insecure \
  --fleet-server-es=http://localhost:9200 \
  --fleet-server-service-token=tokenhere \
  --fleet-server-policy=fleet-server-policy

I succeeded by adding sudo ./elastic-agent install \ --fleet-server-es=https://localhost:9200 \ --fleet-server-service-token=tokenhere \ --fleet-server-policy=fleet-server-policy \ --fleet-server-es-insecure

so changing http -> https and adding --fleet-server-es-insecure at the end

Sources:
@stephenb

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.