Action [indices:admin/auto_create] is unauthorized for API key id [####] of user [elastic/fleet-server] on indices [metricbeat-7.14.1-2021.09.08], this action is granted by the index privileges [auto_configure,create_index,manage,all]

[hamiland]

@Michal_Pristas that seems to have solved it. This has occurred (and now been resolved b the looks) on a brand new installation today on a server that has never had an elastic agent on it. I just checked, the new elastic agent file that is available to download from:

Elastic Agent 7.14.1 | Elastic

In the windows archive the file system.yml not disable in the contained metricbeat archive (I just checked what I had downloaded a couple of hours ago). This may be why @ruflin mentioned that he had a couple of other cases already?

Thank you both so much for bearing with me and for all of the assistance, do we need to do anything else or are we happy this is sorted.

[Michal_Pristas]

if you could just share agent logs it would be helpful.

[hamiland]

Thanks @Michal_Pristas Sorry for being ignorant here, assuming you mean the files from within:

C:\Program Files\Elastic\Agent

that are named elastic-agent-[datecode] with no extension?

[Michal_Pristas]

yes these, thank you,
or they may be in data/elastic-agent-{hash}/logs/elastic-agent-json.log

[hamiland]

And from the freshly installed or one of the older upgraded ones?

[Michal_Pristas]

if freshly installed had the issue we were looking at it will be enough

[hamiland]

Logfile Below

Thanks again @Michal_Pristas

Summary

2021-09-09T14:54:47.183+1000 INFO application/application.go:66 Detecting execution mode
2021-09-09T14:54:47.185+1000 INFO application/application.go:91 Agent is managed by Fleet
2021-09-09T14:54:47.185+1000 INFO capabilities/capabilities.go:59 capabilities file not found in C:\Program Files\Elastic\Agent\capabilities.yml
2021-09-09T14:54:47.186+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T14:54:47.390+1000 INFO [composable.providers.docker] docker/docker.go:43 Docker provider skipped, unable to connect: protocol not available
2021-09-09T14:54:47.390+1000 INFO [api] api/server.go:62 Starting stats endpoint
2021-09-09T14:54:47.390+1000 INFO application/managed_mode.go:291 Agent is starting
2021-09-09T14:54:47.390+1000 INFO [api] api/server.go:64 Metrics endpoint listening on: \.\pipe\elastic-agent (configured: npipe:///elastic-agent)
2021-09-09T14:54:47.754+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T14:54:48.721+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T14:54:48.721+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T14:54:48.721+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T14:54:48.751+1000 INFO stateresolver/stateresolver.go:48 New State ID is WGs8jY6q
2021-09-09T14:54:48.751+1000 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 3 step(s)
2021-09-09T14:54:55.302+1000 INFO log/reporter.go:40 2021-09-09T14:54:55+10:00 - message: Application: filebeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-09T14:54:57.288+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for metricbeat.7.14.1
2021-09-09T14:54:57.520+1000 INFO log/reporter.go:40 2021-09-09T14:54:57+10:00 - message: Application: metricbeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-09T14:54:58.045+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for filebeat.7.14.1
2021-09-09T14:54:58.153+1000 INFO log/reporter.go:40 2021-09-09T14:54:58+10:00 - message: Application: filebeat--7.14.1--36643631373035623733363936343635[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-09T14:54:59.256+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for metricbeat.7.14.1
2021-09-09T14:54:59.364+1000 INFO log/reporter.go:40 2021-09-09T14:54:59+10:00 - message: Application: metricbeat--7.14.1--36643631373035623733363936343635[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-09T14:54:59.372+1000 INFO stateresolver/stateresolver.go:66 Updating internal state
2021-09-09T14:54:59.383+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T14:54:59.594+1000 INFO log/reporter.go:40 2021-09-09T14:54:59+10:00 - message: Application: filebeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-09-09T14:55:00.676+1000 INFO log/reporter.go:40 2021-09-09T14:55:00+10:00 - message: Application: filebeat--7.14.1--36643631373035623733363936343635[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-09-09T14:55:06.306+1000 INFO log/reporter.go:40 2021-09-09T14:55:06+10:00 - message: Application: metricbeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to CONFIG: Updating configuration - type: 'STATE' - sub_type: 'CONFIG'
2021-09-09T14:55:07.334+1000 INFO log/reporter.go:40 2021-09-09T14:55:07+10:00 - message: Application: metricbeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-09-09T14:55:08.038+1000 INFO log/reporter.go:40 2021-09-09T14:55:08+10:00 - message: Application: metricbeat--7.14.1--36643631373035623733363936343635[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
2021-09-09T19:19:34.922+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T19:19:34.922+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-09T19:19:34.957+1000 INFO stateresolver/stateresolver.go:48 New State ID is tE3TE8Gl
2021-09-09T19:19:34.957+1000 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 3 step(s)
2021-09-09T19:19:35.109+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for filebeat.7.14.1
2021-09-09T19:19:35.109+1000 INFO operation/operator.go:260 operation 'operation-start' skipped for filebeat.7.14.1
2021-09-09T19:19:35.286+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for metricbeat.7.14.1
2021-09-09T19:19:35.286+1000 INFO operation/operator.go:260 operation 'operation-start' skipped for metricbeat.7.14.1
2021-09-09T19:19:35.423+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for filebeat.7.14.1
2021-09-09T19:19:35.423+1000 INFO operation/operator.go:260 operation 'operation-start' skipped for filebeat.7.14.1
2021-09-09T19:19:35.562+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for metricbeat.7.14.1
2021-09-09T19:19:35.562+1000 INFO operation/operator.go:260 operation 'operation-start' skipped for metricbeat.7.14.1
2021-09-09T19:19:35.562+1000 INFO stateresolver/stateresolver.go:66 Updating internal state
2021-09-09T22:35:47.257+1000 INFO cmd/run.go:189 Shutting down Elastic Agent and sending last events...
2021-09-09T22:35:47.257+1000 INFO operation/operator.go:192 waiting for installer of pipeline 'default' to finish
2021-09-09T22:35:47.257+1000 INFO process/app.go:176 Signaling application to stop because of shutdown: filebeat--7.14.1
2021-09-09T22:35:47.258+1000 ERROR fleet/fleet_gateway.go:205 Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "https://xxxxxxx:8220/api/fleet/agents/885294a8-4e77-474d-8e1b-a46badfce2b0/checkin?": context canceled
2021-09-09T22:35:57.768+1000 INFO process/app.go:176 Signaling application to stop because of shutdown: metricbeat--7.14.1
2021-09-09T22:35:57.768+1000 INFO log/reporter.go:40 2021-09-09T22:35:57+10:00 - message: Application: filebeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-09T22:36:08.292+1000 INFO process/app.go:176 Signaling application to stop because of shutdown: filebeat--7.14.1--36643631373035623733363936343635
2021-09-09T22:36:08.292+1000 INFO log/reporter.go:40 2021-09-09T22:36:08+10:00 - message: Application: metricbeat--7.14.1[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-09T22:36:18.817+1000 INFO process/app.go:176 Signaling application to stop because of shutdown: metricbeat--7.14.1--36643631373035623733363936343635
2021-09-09T22:36:18.817+1000 INFO log/reporter.go:40 2021-09-09T22:36:18+10:00 - message: Application: filebeat--7.14.1--36643631373035623733363936343635[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-09T22:36:29.340+1000 INFO log/reporter.go:40 2021-09-09T22:36:29+10:00 - message: Application: metricbeat--7.14.1--36643631373035623733363936343635[885294a8-4e77-474d-8e1b-a46badfce2b0]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-09T22:36:29.340+1000 INFO application/managed_mode.go:320 Agent is stopped
2021-09-09T22:36:29.340+1000 INFO cmd/run.go:197 Shutting down completed.
2021-09-09T22:36:29.340+1000 INFO [api] api/server.go:66 Stats endpoint (\.\pipe\elastic-agent) finished: use of closed network connection

[blaker (Blake Rouse)]

Is elasticsearch running on the same host? Because when metricbeat first starts (before Elastic Agent) sends its initial configuration it is possible it might send events to localhost:9200, so that is why I am asking on where is elasticsearch running.

[hamiland]

Hi @blaker elasticsearch is on a different host and this issue was occuring across all of the fleet hosts with the agent policy.

The issue is (and please check this out) that in the official agent download archive system.yml is not system.yml.disabled in the nested metricbeat archive.

@ruflin

but I have now seen this in 2-3 cases already so I assume there is somewhere a bug on our side

This may be for users manually downloading the agent...?

@Michal_Pristas

this here is the problem. we have a hook for metric beat installation and it was for some reason skipped or unsuccessful. please rename system.yml to system.yml.disabled to mitigate the issue. I will go through your logs to see if something is off there

When I had the issue with the new host, I removed the API keys and reset the fleet policy for the rest of the hosts with this policy even though the issue didn't apply to all of them. I then removed and re-installed a fresh elastic agent on all hosts (using the link in the post above). This now meant that all hosts exhibited the same issue until I renamed system.yml back to its disabled form as @Michal_Pristas found.

TLDR; The current elastic agent (Windows at least and re-downloaded just now) for download from: Download Elastic Agent Free is the problem: elastic-agent-7.14.1-windows-x86_64.zip\elastic-agent-7.14.1-windows-x86_64\data\elastic-agent-703d58\downloads\metricbeat-7.14.1-windows-x86_64.zip\metricbeat-7.14.1-windows-x86_64\modules.d\system.yml needs to be renamed to system.yml.disabled

I could be wrong with the above evaluation, however this has resolved the issue for me on 8 hosts.

Thanks again for all of your help @Michal_Pristas, @blaker, @ruflin

[ruflin]

@hamiland Thanks a lot for helping us to debug this issue. We will follow up on our to have a fix in the product for everyone. I filed https://github.com/elastic/beats/issues/27857 to follow up on this.