Action [indices:admin/auto_create] is unauthorized for API key id [####] of user [elastic/fleet-server] on indices [metricbeat-7.14.1-2021.09.08], this action is granted by the index privileges [auto_configure,create_index,manage,all]

hamiland · September 8, 2021, 5:30am

Hi All,
I'm a bit green in the Elastic world so please bear with me. I have Fleet agents deployed to our Windows hosts, however I am having issues when Either "Collect Windows perfmon and service metrics" in the Windows Integration or "Collect Metrics from System instances" is turned on. Having either (or both of these) on results in a flood of events. We're running 7.14.1 and have updated the Fleet integrations - System 1.1.2 and Windows 1.0.0. I have been playing with this for a few hours now and am unsure where I need to go from here. I have removed all tokens and APIs and reinstalled and re-registered all agents. I have reverted the namespace to "default" from some reading I have done that mentioned issues where a namespace other than that may cause issues.

Any suggestions as to where to go from here

blaker · September 8, 2021, 6:23pm

@ruflin Any ideas here? Elastic Agent should not be using the metricbeat-* index at all.

hamiland · September 8, 2021, 10:35pm

Hi @blaker here is the one of the "event details" json message received if this is of any use?:

Summary

{
"_id": "oL-Gx3sBAykVLJegWAeU",
"_index": ".ds-logs-elastic_agent.metricbeat-default-2021.08.20-000005",
"_score": "1",
"_type": "_doc",
"@timestamp": "2021-09-08T22:27:53.683Z",
"agent": {
"ephemeral_id": "25f42c34-bf82-4c92-aa54-ec4e766a249e",
"hostname": "XXXXXXX",
"id": "5e7e18af-79c8-4b04-8b0e-0b453a330572",
"name": "XXXXXX",
"type": "filebeat",
"version": "7.14.1"
},
"data_stream": {
"dataset": "elastic_agent.metricbeat",
"namespace": "default",
"type": "logs"
},
"ecs": {
"version": "1.10.0"
},
"elastic_agent": {
"id": "5e7e18af-79c8-4b04-8b0e-0b453a330572",
"snapshot": "false",
"version": "7.14.1"
},
"event": {
"dataset": "elastic_agent.metricbeat"
},
"host": {
"architecture": "x86_64",
"hostname": "XXXXXX",
"id": "99a8f794-5c97-41d5-80b2-fdfd04239a95",
"ip": "XXXXXX",
"mac": "XXXXXX",
"name": "XXXXXX",
"os": {
"build": "17763.2145",
"family": "windows",
"kernel": "10.0.17763.2145 (WinBuild.160101.0800)",
"name": "Windows Server 2019 Standard",
"platform": "windows",
"type": "windows",
"version": "10.0"
}
},
"input": {
"type": "filestream"
},
"log": {
"level": "warn",
"logger": "elasticsearch",
"offset": "2363",
"origin": {
"file": {
"line": "405",
"name": "elasticsearch/client.go"
}
},
"path": "C:\Program Files\Elastic\Agent\data\elastic-agent-703d58\logs\default\metricbeat-json.log"
},
"message": "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc0466c1a2408d058, ext:58912880251701, loc:(*time.Location)(0x6c50160)}, Meta:null, Fields:{"agent":{"ephemeral_id":"3667b3ac-16f9-405a-8d30-415ba1f66c37","hostname":"XXXXXX","id":"115e7c79-12fb-4019-a61d-c98b6537be67","name":"XXXXXX","type":"metricbeat","version":"7.14.1"},"ecs":{"version":"1.10.0"},"event":{"dataset":"system.process","duration":79533900,"module":"system"},"host":{"architecture":"x86_64","hostname":"XXXXXX","id":"99a8f794-5c97-41d5-80b2-fdfd04239a95","ip":["XXXXXX","XXXXXX"],"mac":["XXXXXX"],"name":"XXXXXX","os":{"build":"17763.2145","family":"windows","kernel":"10.0.17763.2145 (WinBuild.160101.0800)","name":"Windows Server 2019 Standard","platform":"windows","type":"windows","version":"10.0"}},"metricset":{"name":"process","period":10000},"process":{"args":["C:\\Windows\\system32\\lsass.exe"],"command_line":"C:\\Windows\\system32\\lsass.exe","cpu":{"pct":0.001900,"start_time":"2021-09-08T06:05:53.293Z"},"memory":{"pct":0.003600},"name":"lsass.exe","pgid":0,"pid":732,"ppid":580,"state":"running"},"service":{"type":"system"},"system":{"process":{"cmdline":"C:\\Windows\\system32\\lsass.exe","cpu":{"start_time":"2021-09-08T06:05:53.293Z","total":{"norm":{"pct":0.001900},"pct":0.007800,"value":285124.000000}},"memory":{"rss":{"bytes":31023104,"pct":0.003600},"size":16850944},"state":"running"}},"user":{"name":"NT AUTHORITY\\SYSTEM"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [XXXXXX] of user [elastic/fleet-server] on indices [metricbeat-7.14.1-2021.09.09], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}",
"service": {
"name": "metricbeat"
}
}

ruflin · September 9, 2021, 8:33am

Elastic Agent should NOT indexing into metricbeat-* but I have now seen this in 2-3 cases already so I assume there is somewhere a bug on our side. I wonder if it is related to some migration or similar.

@hamiland What is the first version of Elastic Agent and Fleet you have used? Which versions did you migrate through?

The document you put in above, is that all coming from a single log line?

The part I find surprising is that the target index all looks fine "_index": ".ds-logs-elastic_agent.metricbeat-default-2021.08.20-000005", but then it states it can't index into [metricbeat-7.14.1-2021.09.09].

ruflin · September 9, 2021, 8:36am

Update: I think I deciphered the message above. It is the log message event that contains the error as message. And the culprit is an event sent by metricbeat for "dataset":"system.process". It also lacks all the data_stream fields it should contain.

@hamiland Do you have monitoring of Elastic Agent turned on or not? Could you share the yaml content of your policy (without the credentials)?

hamiland · September 9, 2021, 9:25am

Hi @ruflin below is the policy from the fleet console:

I'll post another message with the policy from one of the Agents after this. Most of the settings for the agent is using defaults so (I assume) it's set at info?

This index doesn't appear to exist and the date seems to incrementing daily FYI.

I believe we started with 7.12.0 (That's the earliest metricbeat index) and we use apt to do our version upgrading.

hamiland · September 9, 2021, 9:27am

Policy From Fleet 1/2

id: 5816ce80-103a-11ec-b989-8313176cf53c
revision: 21
outputs:
default:
type: elasticsearch
hosts:
- 'https://XXXXXX:9200'
output_permissions:
default:
system:
indices:
- names:
- logs-system.auth-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.syslog-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.application-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.security-default
privileges:
- auto_configure
- create_doc
- names:
- logs-system.system-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.fsstat-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process.summary-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.core-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.diskio-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.filesystem-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.cpu-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.socket_summary-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.process-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.load-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.network-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.uptime-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-system.memory-default
privileges:
- auto_configure
- create_doc
windows:
indices:
- names:
- logs-windows.forwarded-default
privileges:
- auto_configure
- create_doc
- names:
- logs-windows.powershell-default
privileges:
- auto_configure
- create_doc
- names:
- logs-windows.powershell_operational-default
privileges:
- auto_configure
- create_doc
- names:
- logs-windows.sysmon_operational-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-windows.perfmon-default
privileges:
- auto_configure
- create_doc
- names:
- metrics-windows.service-default
privileges:
- auto_configure
- create_doc
_elastic_agent_checks:
cluster:
- monitor
indices:
- names:
- logs-elastic_agent-default
- logs-elastic_agent.elastic_agent-default
- logs-elastic_agent.apm_server-default
- logs-elastic_agent.filebeat-default
- logs-elastic_agent.fleet_server-default
- logs-elastic_agent.metricbeat-default
- logs-elastic_agent.osquerybeat-default
- logs-elastic_agent.packetbeat-default
- logs-elastic_agent.endpoint_security-default
- logs-elastic_agent.auditbeat-default
- logs-elastic_agent.heartbeat-default
- metrics-elastic_agent-default
- metrics-elastic_agent.elastic_agent-default
- metrics-elastic_agent.apm_server-default
- metrics-elastic_agent.filebeat-default
- metrics-elastic_agent.fleet_server-default
- metrics-elastic_agent.metricbeat-default
- metrics-elastic_agent.osquerybeat-default
- metrics-elastic_agent.packetbeat-default
- metrics-elastic_agent.endpoint_security-default
- metrics-elastic_agent.auditbeat-default
- metrics-elastic_agent.heartbeat-default
privileges:
- auto_configure
- create_doc
agent:
monitoring:
enabled: true
use_output: default
namespace: default
logs: true
metrics: true

hamiland · September 9, 2021, 9:28am

Policy from Fleet 2/2

inputs:
  - id: f0385cd8-92b5-4edb-86e2-4f0d639292e8
    name: system
    revision: 8
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 1.1.2
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.auth
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              fields:
                ecs.version: 1.9.0
              target: ''
      - id: logfile-system.syslog-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.syslog
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              fields:
                ecs.version: 1.9.0
              target: ''
  - id: f0385cd8-92b5-4edb-86e2-4f0d639292e8
    name: system
    revision: 8
    type: winlog
    use_output: default
    meta:
      package:
        name: system
        version: 1.1.2
    data_stream:
      namespace: default
    streams:
      - id: winlog-system.application-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        name: Application
        data_stream:
          dataset: system.application
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
        tags: null
      - id: winlog-system.security-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        name: Security
        data_stream:
          dataset: system.security
          type: logs
        condition: '${host.platform} == ''windows'''
        tags: null
      - id: winlog-system.system-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        name: System
        data_stream:
          dataset: system.system
          type: logs
        condition: '${host.platform} == ''windows'''
        tags: null
  - id: f0385cd8-92b5-4edb-86e2-4f0d639292e8
    name: system
    revision: 8
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 1.1.2
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.fsstat-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.fsstat
          type: metrics
        period: 1m
        metricsets:
          - fsstat
        processors:
          - drop_event.when.regexp:
              system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: >-
          system/metrics-system.process.summary-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.process.summary
          type: metrics
        period: 10s
        metricsets:
          - process_summary
      - id: system/metrics-system.core-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.core
          type: metrics
        metricsets:
          - core
        core.metrics:
          - percentages
      - id: system/metrics-system.diskio-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.diskio
          type: metrics
        period: 10s
        diskio.include_devices: null
        metricsets:
          - diskio
      - id: system/metrics-system.filesystem-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.filesystem
          type: metrics
        period: 1m
        metricsets:
          - filesystem
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.cpu-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.cpu
          type: metrics
        period: 10s
        cpu.metrics:
          - percentages
          - normalized_percentages
        metricsets:
          - cpu
      - id: >-
          system/metrics-system.socket_summary-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.socket_summary
          type: metrics
        period: 10s
        metricsets:
          - socket_summary
      - id: system/metrics-system.process-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.process
          type: metrics
        process.include_top_n.by_memory: 5
        period: 10s
        processes:
          - .*
        process.include_top_n.by_cpu: 5
        process.cgroups.enabled: false
        process.cmdline.cache.enabled: true
        metricsets:
          - process
        process.include_cpu_ticks: false
      - id: system/metrics-system.load-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.load
          type: metrics
        condition: '${host.platform} != ''windows'''
        period: 10s
        metricsets:
          - load
      - id: system/metrics-system.network-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.network
          type: metrics
        period: 10s
        network.interfaces: null
        metricsets:
          - network
      - id: system/metrics-system.uptime-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.uptime
          type: metrics
        period: 10s
        metricsets:
          - uptime
      - id: system/metrics-system.memory-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.memory
          type: metrics
        period: 10s
        metricsets:
          - memory
  - id: 288812d7-cd12-42b7-a816-c3a45a446e20
    name: windows
    revision: 9
    type: winlog
    use_output: default
    meta:
      package:
        name: windows
        version: 1.0.0
    data_stream:
      namespace: default
    streams:
      - id: winlog-windows.forwarded-288812d7-cd12-42b7-a816-c3a45a446e20
        name: ForwardedEvents
        data_stream:
          dataset: windows.forwarded
          type: logs
        condition: '${host.platform} == ''windows'''
        tags:
          - forwarded
        publisher_pipeline.disable_host: true
      - id: winlog-windows.powershell-288812d7-cd12-42b7-a816-c3a45a446e20
        name: Windows PowerShell
        data_stream:
          dataset: windows.powershell
          type: logs
        condition: '${host.platform} == ''windows'''
        event_id: '400, 403, 600, 800'
      - id: >-
          winlog-windows.powershell_operational-288812d7-cd12-42b7-a816-c3a45a446e20
        name: Microsoft-Windows-PowerShell/Operational
        data_stream:
          dataset: windows.powershell_operational
          type: logs
        condition: '${host.platform} == ''windows'''
        event_id: '4103, 4104, 4105, 4106'
      - id: winlog-windows.sysmon_operational-288812d7-cd12-42b7-a816-c3a45a446e20
        name: Microsoft-Windows-Sysmon/Operational
        data_stream:
          dataset: windows.sysmon_operational
          type: logs
        condition: '${host.platform} == ''windows'''
  - id: 288812d7-cd12-42b7-a816-c3a45a446e20
    name: windows
    revision: 9
    type: windows/metrics
    use_output: default
    meta:
      package:
        name: windows
        version: 1.0.0
    data_stream:
      namespace: default
    streams:
      - id: windows/metrics-windows.perfmon-288812d7-cd12-42b7-a816-c3a45a446e20
        data_stream:
          dataset: windows.perfmon
          type: metrics
        condition: '${host.platform} == ''windows'''
        period: 10s
        perfmon.ignore_non_existent_counters: true
        metricsets:
          - perfmon
        perfmon.queries:
          - instance:
              - '*'
            counters:
              - name: '% Processor Time'
                field: cpu_perc
                format: float
              - name: Working Set
            object: Process
        perfmon.group_measurements_by_instance: true
      - id: windows/metrics-windows.service-288812d7-cd12-42b7-a816-c3a45a446e20
        data_stream:
          dataset: windows.service
          type: metrics
        condition: '${host.platform} == ''windows'''
        period: 60s
        metricsets:
          - service
fleet:
  hosts:
    - 'https://XXXXXX:8220'

ruflin · September 9, 2021, 9:37am

The policy part looks all good. I was looking for this bit here:

     - id: system/metrics-system.process-f0385cd8-92b5-4edb-86e2-4f0d639292e8
        data_stream:
          dataset: system.process
          type: metrics
        process.include_top_n.by_memory: 5
        period: 10s
        processes:
          - .*
        process.include_top_n.by_cpu: 5
        process.cgroups.enabled: false
        process.cmdline.cache.enabled: true
        metricsets:
          - process
        process.include_cpu_ticks: false

I was suspecting that it might miss the data_stream.* settings but it doesn't.

This is a bit out in the blue but worth a try: Can you stop elastic-agent and check if you have still a metricbeat instance running somewhere on the machine? I assume you are using only Elastic Agent? If there is no metricbeat instance running after Elastic agent is stopped, start it again and see if the error still show up.

Which Windows OS are you on?

hamiland · September 9, 2021, 9:42am

Hi @ruflin, I stopped the Elastic agent and the 2 instances of metricbeat.exe stopped, so doesn't appear to have resolved it (and the same errors are still being logged unfortunately). I have tried restarts and that doesn't appear to resolve the issue. We are running Windows server 2019.

I assume I retrieved the config that you needed or do you need more?

Do you need any logging, if so let me know what you need or how to enable and gather for you?

hamiland · September 9, 2021, 10:20am

Sorry, I didn't give you the versions that we have upgraded through:

7.12.0
7.12.1
7.13.0
7.13.3
7.14.0
7.14.1

Thanks

Michal_Pristas · September 9, 2021, 10:32am

hey @hamiland we're having difficulties reproducing this but I have a slight suspicion.
is it possible for you to try to restart a machine for me and check if it goes away please?

hamiland · September 9, 2021, 10:48am

Hi @Michal_Pristas, I have just restarted 2 of the servers, and the issues remain with the same "Cannot Index event publisher" messages for both.

ruflin · September 9, 2021, 11:22am

Can you share the output of elastic-agent inspect output -o default -p metricbeat ?

hamiland · September 9, 2021, 11:28am

Hi @ruflin as requested, hopefully there's something obvious in there.....

Thanks for this.

Output of elastic-agent inspect output -o default -p metricbeat

c:\Program Files\Elastic\Agent>elastic-agent inspect output -o default -p metricbeat

[default] metricbeat:
metricbeat:
  modules:
  - id: system/metrics-system.fsstat-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.fsstat-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - fsstat
    module: system
    name: system
    period: 1m
    processors:
    - drop_event:
        when:
          regexp:
            system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
    - add_fields:
        fields:
          dataset: system.fsstat
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.fsstat
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.process.summary-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.process.summary-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - process_summary
    module: system
    name: system
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.process.summary
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.process.summary
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - core.metrics:
    - percentages
    id: system/metrics-system.core-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.core-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - core
    module: system
    name: system
    processors:
    - add_fields:
        fields:
          dataset: system.core
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.core
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - diskio.include_devices: null
    id: system/metrics-system.diskio-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.diskio-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - diskio
    module: system
    name: system
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.diskio
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.diskio
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.filesystem-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.filesystem-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - filesystem
    module: system
    name: system
    period: 1m
    processors:
    - drop_event:
        when:
          regexp:
            system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
    - add_fields:
        fields:
          dataset: system.filesystem
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.filesystem
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - cpu.metrics:
    - percentages
    - normalized_percentages
    id: system/metrics-system.cpu-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.cpu-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - cpu
    module: system
    name: system
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.cpu
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.cpu
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.socket_summary-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.socket_summary-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - socket_summary
    module: system
    name: system
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.socket_summary
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.socket_summary
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.process-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.process-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - process
    module: system
    name: system
    period: 10s
    process.cgroups.enabled: false
    process.cmdline.cache.enabled: true
    process.include_cpu_ticks: false
    process.include_top_n.by_cpu: 5
    process.include_top_n.by_memory: 5
    processes:
    - .*
    processors:
    - add_fields:
        fields:
          dataset: system.process
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.process
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.network-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.network-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - network
    module: system
    name: system
    network.interfaces: null
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.network
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.network
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.uptime-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.uptime-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - uptime
    module: system
    name: system
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.uptime
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.uptime
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: system/metrics-system.memory-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.memory-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - memory
    module: system
    name: system
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: system.memory
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.memory
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8
  - id: windows/metrics-windows.perfmon-288812d7-cd12-42b7-a816-c3a45a446e20
    index: metrics-windows.perfmon-default
    meta:
      package:
        name: windows
        version: 1.0.0
    metricsets:
    - perfmon
    module: windows
    name: windows
    perfmon.group_measurements_by_instance: true
    perfmon.ignore_non_existent_counters: true
    perfmon.queries:
    - counters:
      - field: cpu_perc
        format: float
        name: '% Processor Time'
      - name: Working Set
      instance:
      - '*'
      object: Process
    period: 10s
    processors:
    - add_fields:
        fields:
          dataset: windows.perfmon
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: windows.perfmon
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 9
  - id: windows/metrics-windows.service-288812d7-cd12-42b7-a816-c3a45a446e20
    index: metrics-windows.service-default
    meta:
      package:
        name: windows
        version: 1.0.0
    metricsets:
    - service
    module: windows
    name: windows
    period: 60s
    processors:
    - add_fields:
        fields:
          dataset: windows.service
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: windows.service
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 9
output:
  elasticsearch:
    api_key: XXXXXX
    hosts:
    - https://XXXXXX:9200

ruflin · September 9, 2021, 11:48am

It does not contain what I hoped for Here the the system.process part from the config:

  - id: system/metrics-system.process-f0385cd8-92b5-4edb-86e2-4f0d639292e8
    index: metrics-system.process-default
    meta:
      package:
        name: system
        version: 1.1.2
    metricsets:
    - process
    module: system
    name: system
    period: 10s
    process.cgroups.enabled: false
    process.cmdline.cache.enabled: true
    process.include_cpu_ticks: false
    process.include_top_n.by_cpu: 5
    process.include_top_n.by_memory: 5
    processes:
    - .*
    processors:
    - add_fields:
        fields:
          dataset: system.process
          namespace: default
          type: metrics
        target: data_stream
    - add_fields:
        fields:
          dataset: system.process
        target: event
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
          snapshot: false
          version: 7.14.1
        target: elastic_agent
    - add_fields:
        fields:
          id: 04e4dac5-1374-4a8b-9407-3f793e915fd0
        target: agent
    revision: 8

The error you have would indicate the index part is missing or wrong. But as we can see, it is there.

Now I wonder if there might be more outputs. Can you run the following command?

elastic-agent inspect output

In case you see more then just default can you run the previous command with it?

@michalpristas Other ideas on your end?

@hamiland: Hope you don't mind that I formatted the output you posted a bit so the indentation is visible.

hamiland · September 9, 2021, 11:51am

@ruflin bad news I'm afraid.....

ruflin:

Now I wonder if there might be more outputs. Can you run the following command?
elastic-agent inspect output
In case you see more then just default can you run the previous command with it?

c:\Program Files\Elastic\Agent>elastic-agent inspect output
default

c:\Program Files\Elastic\Agent>

I assume that the blank line is there as part of the return and not another output that is unnamed?

Michal_Pristas · September 9, 2021, 12:22pm

can you list files in data/elastic-agent-{hash}/install/metricbeat-*/modules.d/

hamiland · September 9, 2021, 12:27pm

Thanks @Michal_Pristas, below is the contents of the folder requested,

File Listing

Directory of c:\Program Files\Elastic\Agent\data\elastic-agent-703d58\install\metricbeat-7.14.1-windows-x86_64\modules.d

08/09/2021 01:32 PM .
08/09/2021 01:32 PM ..
08/09/2021 01:32 PM 355 activemq.yml.disabled
08/09/2021 01:32 PM 209 aerospike.yml.disabled
08/09/2021 01:32 PM 241 apache.yml.disabled
08/09/2021 01:32 PM 271 appsearch.yml.disabled
08/09/2021 01:32 PM 1,032 aws.yml.disabled
08/09/2021 01:32 PM 183 awsfargate.yml.disabled
08/09/2021 01:32 PM 3,219 azure.yml.disabled
08/09/2021 01:32 PM 234 beat-xpack.yml.disabled
08/09/2021 01:32 PM 208 beat.yml.disabled
08/09/2021 01:32 PM 484 ceph-mgr.yml.disabled
08/09/2021 01:32 PM 424 ceph.yml.disabled
08/09/2021 01:32 PM 383 cloudfoundry.yml.disabled
08/09/2021 01:32 PM 361 cockroachdb.yml.disabled
08/09/2021 01:32 PM 209 consul.yml.disabled
08/09/2021 01:32 PM 195 coredns.yml.disabled
08/09/2021 01:32 PM 275 couchbase.yml.disabled
08/09/2021 01:32 PM 196 couchdb.yml.disabled
08/09/2021 01:32 PM 661 docker.yml.disabled
08/09/2021 01:32 PM 308 dropwizard.yml.disabled
08/09/2021 01:32 PM 261 elasticsearch-xpack.yml.disabled
08/09/2021 01:32 PM 283 elasticsearch.yml.disabled
08/09/2021 01:32 PM 209 envoyproxy.yml.disabled
08/09/2021 01:32 PM 273 etcd.yml.disabled
08/09/2021 01:32 PM 1,321 gcp.yml.disabled
08/09/2021 01:32 PM 295 golang.yml.disabled
08/09/2021 01:32 PM 458 graphite.yml.disabled
08/09/2021 01:32 PM 217 haproxy.yml.disabled
08/09/2021 01:32 PM 653 http.yml.disabled
08/09/2021 01:32 PM 335 ibmmq.yml.disabled
08/09/2021 01:32 PM 280 iis.yml.disabled
08/09/2021 01:32 PM 1,874 istio.yml.disabled
08/09/2021 01:32 PM 981 jolokia.yml.disabled
08/09/2021 01:32 PM 1,563 kafka.yml.disabled
08/09/2021 01:32 PM 248 kibana-xpack.yml.disabled
08/09/2021 01:32 PM 255 kibana.yml.disabled
08/09/2021 01:32 PM 1,347 kubernetes.yml.disabled
08/09/2021 01:32 PM 228 kvm.yml.disabled
08/09/2021 01:32 PM 264 linux.yml.disabled
08/09/2021 01:32 PM 238 logstash-xpack.yml.disabled
08/09/2021 01:32 PM 261 logstash.yml.disabled
08/09/2021 01:32 PM 203 memcached.yml.disabled
08/09/2021 01:32 PM 1,170 mongodb.yml.disabled
08/09/2021 01:32 PM 290 mssql.yml.disabled
08/09/2021 01:32 PM 192 munin.yml.disabled
08/09/2021 01:32 PM 738 mysql.yml.disabled
08/09/2021 01:32 PM 499 nats.yml.disabled
08/09/2021 01:32 PM 347 nginx.yml.disabled
08/09/2021 01:32 PM 409 openmetrics.yml.disabled
08/09/2021 01:32 PM 293 oracle.yml.disabled
08/09/2021 01:32 PM 280 php_fpm.yml.disabled
08/09/2021 01:32 PM 290 postgresql.yml.disabled
08/09/2021 01:32 PM 2,249 prometheus.yml.disabled
08/09/2021 01:32 PM 271 rabbitmq.yml.disabled
08/09/2021 01:32 PM 441 redis.yml.disabled
08/09/2021 01:32 PM 262 redisenterprise.yml.disabled
08/09/2021 01:32 PM 309 sql.yml.disabled
08/09/2021 01:32 PM 454 stan.yml.disabled
08/09/2021 01:32 PM 192 statsd.yml.disabled
08/09/2021 01:32 PM 960 system.yml
08/09/2021 01:32 PM 287 tomcat.yml.disabled
08/09/2021 01:32 PM 196 traefik.yml.disabled
08/09/2021 01:32 PM 200 uwsgi.yml.disabled
08/09/2021 01:32 PM 478 vsphere.yml.disabled
08/09/2021 01:32 PM 426 windows.yml.disabled
08/09/2021 01:32 PM 218 zookeeper.yml.disabled
65 File(s) 32,946 bytes
2 Dir(s) 97,630,019,584 bytes free

and just in case I'll try and pre-empt your next request given it's the only file without "disabled"....

system.yml

Module: system

Docs: System module | Metricbeat Reference [7.x] | Elastic

module: system
period: 10s
metricsets:
- cpu
  #- load
- memory
- network
- process
- process_summary
- socket_summary
  #- entropy
  #- core
  #- diskio
  #- socket
  #- service
  #- users
  process.include_top_n:
  by_cpu: 5 # include top 5 processes by CPU
  by_memory: 5 # include top 5 processes by memory
Configure the mount point of the hostÔÇÖs filesystem for use in monitoring a host from within a container

#system.hostfs: "/hostfs"
module: system
period: 1m
metricsets:
- filesystem
- fsstat
  processors:
- drop_event.when.regexp:
  system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
module: system
period: 15m
metricsets:
- uptime

#- module: system

period: 5m

metricsets:

- raid

raid.mount_point: '/'

Sorry about the contents of system, the forum has applied some really funky formatting there.

Michal_Pristas · September 9, 2021, 12:29pm

system.yml
this here is the problem. we have a hook for metric beat installation and it was for some reason skipped or unsuccessful. please rename system.yml to system.yml.disabled to mitigate the issue. I will go through your logs to see if something is off there

you might need an agent restart after rename