Agent service will not start, failure to authenticate to Kibana when using API key w/Elasticsearch

[dvo]

Hello all, we're doing a POC with Windows servers running *beat agents sending perf and event log data to a Linux VM running ELK Stack. Everything is latest version. We want to standardize on agents authenticating using API key, and don't see much need to go beyond out of the box user roles.

Metricbeat agent authenticates fine when metricbeat.yml uses the builtin elastic superuser account/password. Great. I then created an API key with no role defined while logged in as elastic. Documentation states the key should effectively have rights of the user creating it, so superuser rights. A bit much, but I just need to see it work.

After editing Elasticsearch.yml to use the API key, the agent service starts/stops. The error is a 401, failure to authenticate to Kibana. During troubleshooting, discovered I can work around this by adding the elastic username and password under the setup.kibana section. That is obviously not secure and unusable. I can't find documentation showing where you must use an Elasticsearch user/password under setup.kibana in order to authenticate to Kibana when choosing to use API key to authenticate with Beats. I also don't see any kind of API key setting for Kibana in kibana.yml

Advice is greatly appreciated, and thank you in advance!

[stephenb (Stephen Brown)]

Hi @dvo

I am a little unclear.. can you clarify if you are using metricbeat or the New Elastic Agent Via Fleet?

Also when you refer to editing the elasticsearch.yml did you mean the metricbeat.yml? because you configure metricbeat with the metricbeat.yml.

Assuming you mean the metricbeat.yml

Interesting....I need to look closer but Kibana setup config and / creds is only used during metricbeat setup phase, which only needs to be run once. So for normal running you can just comment out that setup.kibana section

Let us know..

[dvo]

Thanks for replying!

Please forgive my mistake, I did mean metricbeat.yml for the 7.10.2 metricbeat Windows agent.

I too was thinking I could simply remark out the Kibana section of metricbeat.yml and wondered why that did not work. I overlooked remarking out setup.dashboards.enabled: true. Once I did that, the agent remained running and all appears well.

Bravo for moving towards a multipurpose agent that can be (somewhat) centrally managed! Prior to retiring I had 15 years with MOM/SCOM and Azure Monitor/Log Analytics for about 20,000 on-prem servers. Have been understandably lamenting multiple agents/multiple configs despite appreciating the small footprint. Hopefully Fleet will go GA soon so it can be fully evaluated.

Thank you Stephen!

[stephenb (Stephen Brown)]

Yes Fleet is pretty ambitious for us, it will go GA in a bit, eventually I understand it will be Policy Driven (today) API, Terraform provider etc driven.

Glad you got it working in the mean time.