Precondition 'apm integration installed' failed

Elastic Observability APM
1

Hello, i am trying to setup apm server (legacy). I successfully deployed apm server with eck to kubernetes. But when i check logs i see error message related to integration. I created integration from kibana UI, but still getting the same error.

apm1
apm13284×970 148 KB

And i don't see any data in kibana

apm2
apm22204×1452 184 KB

I see this index templates created

apm_indexes
apm_indexes2986×1424 403 KB

But don't see any data stream created. APM config:

apm-server:
  host: :8200
  kibana:
    enabled: true
    host: https://kibana
    password: pass
    ssl:
      certificate_authorities:
      - config/kibana-certs/ca.crt
    username: elastic-common-cluster-apm-kb-user
  secret_token: ${SECRET_TOKEN}
  ssl:
    certificate: /mnt/elastic-internal/http-certs/tls.crt
    enabled: true
    key: /mnt/elastic-internal/http-certs/tls.key
logging:
  level: debug
output:
  logstash:
    hosts:
    - logstash:5044
    index: apm-server
setup:
  template:
    enabled: true

and logstash config for apm

filter {
  if [@metadata][beat] == "apm"{
    if [processor][event] == "sourcemap" {
      mutate {
        add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" }
      }
    } else {
      mutate {
        add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" }
      }
    }
  }
}

output {
  if [agent][type] == "filebeat" {
    elasticsearch {
      hosts => ["elastic9200"]
      index => "app-%{[logtype]}-%{[@metadata][ds_name]}"
      ilm_enabled => false
      action => create
      ssl => true
      cacert => "/usr/share/elasticsearch/config/http-certs/ca.crt"
      user => "${ELASTICSEARCH_USERNAME}"
      password => "${ELASTICSEARCH_PASSWORD}"
    }
  } else if [agent][type] == "metricbeat" or [agent][type] == "auditbeat" or [agent][type] == "heartbeat" or [@metadata][beat] == "apm" {
    elasticsearch {
      hosts => ["elastic:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      ilm_enabled => false
      action => create
      ssl => true
      cacert => "/usr/share/elasticsearch/config/http-certs/ca.crt"
      user => "${ELASTICSEARCH_USERNAME}"
      password => "${ELASTICSEARCH_PASSWORD}"
    }
}

Kibana version: 8.4.1

Elasticsearch version:8.4.1

APM Server version:8.4.1

APM Agent language and version: opentelemtry, postman

Browser version: latest chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version: eck operator

Fresh install or upgraded from other version? both

Is there anything special in your setup? Logstash as output

Provide logs and/or server output (if relevant):

{"log.level":"error","@timestamp":"2022-09-14T08:23:35.643Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Kibana for integration package status: unexpected HTTP status: 403 Forbidden ({\"statusCode\":403,\"error\":\"Forbidden\",\"message\":\"Forbidden\"}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}
2

The suggested remediation action ("please install the apm integration") is a misleading in this case. The issue appears to be that your Kibana credentials do not allow APM Server to query the installed integration packages.

You can either expand the privileges of elastic-common-cluster-apm-kb-user, or you can set apm-server.data_streams.wait_for_integration: false in your APM Server configuration, which will cause APM Server to index data without first waiting for the integration to be installed.

3

I switched to using elasticsearch as output and issue gone

4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.