Error Elastic APM Self Hosting

kelvinferraz · May 16, 2022, 2:43am

Hello everyone!

Recently I decided to test the version of 8.2 for my stack Elastic, including APM. I was using the 7.5 version.

But I found the serious problems to understand and configure the self-hosted version.

In this scenario, I do the configuration of the complete stack, including the APM again, but when I tried to test, my scenario don't work very well, like my APM received some erros like:


[elastic_agent.apm_server][error] failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for API key id [xB6myoAB0eENg0fBank6] of user [elastic/fleet-server] on indices [metrics-apm.internal-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]

I don't know if I made the correct configuration, but for some reason, my script test can't show me something in the monitoring dashboard of Kibana (just that I already configured the fleet in the environment).

APM Status:

Fleet status:

Kibana version: 8.2

Elasticsearch version:8.2

APM Server version: (I don't know exactly what this use, because after configured my fleet, the server stop to start, I believe is 8.2 too)

I will be grateful for any help.

Regards.

kelvinferraz · May 30, 2022, 5:16pm

Hello someone can help me?

simitt · May 30, 2022, 6:06pm

Hi @kelvinferraz could you share the Elastic Agent policy that you are using (please redact any sensitive information before sharing).
You will find the policy when navigating to Fleet/Agent policies/ clicking on the Actions button and select View policies, similar to:

kelvinferraz · May 30, 2022, 6:28pm

Hello simitt,

simitt · May 31, 2022, 5:52am

What you shared is just the overview of the installed integrations, please click on Actions/View Policy and share the policy.

kelvinferraz · May 31, 2022, 11:26am

So sorry,

My Policy:

id: 7e703890-d1f1-11ec-b247-a942f89349a3
revision: 10
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://MYIPADDRESS:9200'
    ssl.ca_trusted_fingerprint: 89ac986af82ab7b31da528d5bb655621e53fbad63de339636f568905372f4308
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    system-3:
      indices:
        - names:
            - logs-system.auth-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.syslog-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.application-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.security-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.system-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.cpu-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.diskio-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.filesystem-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.fsstat-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.load-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.memory-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.network-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.process-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.process.summary-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.socket_summary-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.uptime-production
          privileges:
            - auto_configure
            - create_doc
    apm-server:
      indices:
        - names:
            - logs-apm.app-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-apm.app.*-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-apm.error-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-apm.internal-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-apm.profiling-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - traces-apm.rum-production
          privileges:
            - auto_configure
            - create_doc
        - names:
            - traces-apm.sampled-production
          privileges:
            - auto_configure
            - create_doc
            - maintenance
            - monitor
            - read
        - names:
            - traces-apm-production
          privileges:
            - auto_configure
            - create_doc
      cluster:
        - 'cluster:monitor/main'
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: logfile-system-bd988dae-3733-4715-81a2-2a6bb0a39505
    name: system-3
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 1.11.0
    data_stream:
      namespace: production
    streams:
      - id: logfile-system.auth-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.auth
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
      - id: logfile-system.syslog-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.syslog
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
  - id: winlog-system-bd988dae-3733-4715-81a2-2a6bb0a39505
    name: system-3
    revision: 2
    type: winlog
    use_output: default
    meta:
      package:
        name: system
        version: 1.11.0
    data_stream:
      namespace: production
    streams:
      - id: winlog-system.application-bd988dae-3733-4715-81a2-2a6bb0a39505
        name: Application
        data_stream:
          dataset: system.application
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
      - id: winlog-system.security-bd988dae-3733-4715-81a2-2a6bb0a39505
        name: Security
        data_stream:
          dataset: system.security
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
      - id: winlog-system.system-bd988dae-3733-4715-81a2-2a6bb0a39505
        name: System
        data_stream:
          dataset: system.system
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
  - id: system/metrics-system-bd988dae-3733-4715-81a2-2a6bb0a39505
    name: system-3
    revision: 2
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 1.11.0
    data_stream:
      namespace: production
    streams:
      - id: system/metrics-system.cpu-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.cpu
          type: metrics
        period: 10s
        cpu.metrics:
          - percentages
          - normalized_percentages
        metricsets:
          - cpu
      - id: system/metrics-system.diskio-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.diskio
          type: metrics
        period: 10s
        diskio.include_devices: null
        metricsets:
          - diskio
      - id: system/metrics-system.filesystem-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.filesystem
          type: metrics
        period: 1m
        metricsets:
          - filesystem
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.fsstat-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.fsstat
          type: metrics
        period: 1m
        metricsets:
          - fsstat
        processors:
          - drop_event.when.regexp:
              system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.load-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.load
          type: metrics
        condition: '${host.platform} != ''windows'''
        period: 10s
        metricsets:
          - load
      - id: system/metrics-system.memory-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.memory
          type: metrics
        period: 10s
        metricsets:
          - memory
      - id: system/metrics-system.network-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.network
          type: metrics
        period: 10s
        network.interfaces: null
        metricsets:
          - network
      - id: system/metrics-system.process-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.process
          type: metrics
        process.include_top_n.by_memory: 5
        period: 10s
        processes:
          - .*
        process.include_top_n.by_cpu: 5
        process.cgroups.enabled: false
        process.cmdline.cache.enabled: true
        metricsets:
          - process
        process.include_cpu_ticks: false
      - id: >-
          system/metrics-system.process.summary-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.process.summary
          type: metrics
        period: 10s
        metricsets:
          - process_summary
      - id: >-
          system/metrics-system.socket_summary-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.socket_summary
          type: metrics
        period: 10s
        metricsets:
          - socket_summary
      - id: system/metrics-system.uptime-bd988dae-3733-4715-81a2-2a6bb0a39505
        data_stream:
          dataset: system.uptime
          type: metrics
        period: 10s
        metricsets:
          - uptime
  - id: 318b41c2-4297-4f4e-973d-742376b03207
    name: apm-server
    revision: 4
    type: apm
    use_output: default
    meta:
      package:
        name: apm
        version: 8.2.0
    data_stream:
      namespace: production
    apm-server:
      capture_personal_data: true
      max_connections: 0
      max_event_size: 307200
      auth:
        api_key:
          enabled: true
          limit: 100
        anonymous:
          enabled: false
          allow_agent:
            - rum-js
            - js-base
            - iOS/swift
          allow_service: null
          rate_limit:
            ip_limit: 1000
            event_limit: 300
        secret_token: >-
          MzYyY2IyNzhkMjQwMGFmMjE0Y2NiZGI2MTMzZTc0ZDdkMzQyNWI0ZjZiZDNlZmUwZGFlMzVlOGQ4
          M2E3MTQyZiAgLQo=
      default_service_environment: null
      shutdown_timeout: 30s
      sampling:
        tail:
          enabled: false
          policies:
            - sample_rate: 0.1
          interval: 1m
      rum:
        enabled: true
        exclude_from_grouping: ^/webpack
        allow_headers: null
        response_headers: null
        library_pattern: node_modules|bower_components|~
        allow_origins:
          - '*'
        source_mapping:
          metadata: []
      ssl:
        enabled: false
        key_passphrase: null
        certificate: null
        supported_protocols:
          - TLSv1.0
          - TLSv1.1
          - TLSv1.2
        curve_types: null
        key: null
        cipher_suites: null
      response_headers: null
      write_timeout: 30s
      host: 'MYIPADDRESS:8200'
      max_header_size: 1048576
      idle_timeout: 45s
      expvar.enabled: false
      read_timeout: 3600s
      java_attacher:
        enabled: false
        discovery-rules: null
        download-agent-version: null
      agent_config:
        - service:
            name: ENVIRONMENT_ALL
          etag: 84e01191de16e92ec80eb39d5dc3524fc396a46a
          config:
            transaction_max_spans: '500'
            transaction_sample_rate: '1.0'
fleet:
  hosts:
    - 'https://MYIPADDRESS:8220'
    - 'https://MYIPADDRESS:8220'
    - 'https://MYIPADDRESS:8220'
    - 'https://MYIPADDRESS:8220'
    - 'https://MYIPADDRESS:8220'
    - 'https://MYIPADDRESS:8220'
    - 'https://localhost:8220'
    - 'https://localhost:8220'

simitt · May 31, 2022, 2:39pm

The policy shows that you have configured a custom namespace for apm:

data_stream:
      namespace: production

Unfortunately there was a bug in 8.2, breaking ingestion for custom namespaces (apm-server#8087). The bug is fixed and the fix will be released with the next version. Until then, please switch back the namespace to default and data ingestion should work as expected again.

kelvinferraz · May 31, 2022, 3:20pm

This information is great to know.

How I can change this configuration to default?

Thanks agan.

simitt · May 31, 2022, 5:38pm

When changing via UI you navigate to the elastic agent policy, click on the apm integration which will open it for editing. One of the first attributes is the namespace. Remove the production and replace by default. Then save the changes.

By default the namespace is set to default, so someone from your organization must have changed this. If they applied the change via automation or tooling, I suggest you directly update it there.

system · June 21, 2022, 1:39pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.