Default APM Server docker did not create apm-7.5.1-transaction and apm-7.5.1-span with suffix number behind

Kibana version: 7.5.1

Elasticsearch version: 7.5.1

APM Server version: 7.5.1

APM Agent language and version: go

Browser version: Version 79.0.3945.88 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker image apm-server:7.5.1

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup?
We're using fluent bit to ship log to Elasticsearch

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
After install apm-server and access to output Elasticsearch, checked the Indice I don't see the indices had been created with "apm-7.5.1-transaction-000001" and "apm-7.5.1-span-000001". That make the life cycle rotate did not work for two indices above. Error generated:
illegal_argument_exception: index.lifecycle.rollover_alias [apm-7.5.1-transaction] does not point to index [apm-7.5.1-transaction]

⎈ |PRODUCTION_USE1:logging)]$ curl   localhost:9200/_cat/indices?v | grep span
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 9200
100  4953  100  4953    0     0   3984      0  0:00:01  0:00:01 --:--:--  3987
green  open   apm-7.5.1-span                  FnJJgBSUTOKpmRf3mux1yg   1   1   43516421            0     25.8gb         12.9gb

Steps to reproduce:

  1. Install apm as K8s DaemonSet
  2. Verify that apm can output to ES successfully
  3. Check the indices make sure ES has Indices with suffix apm-7.5.1-transaction-000001

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

apiVersion: apps/v1
kind: DaemonSet
  name: euw1-apm
  namespace: logging
    app: euw1-apm
    apm-infra: euw1-apm
    type: RollingUpdate
      maxUnavailable: 100
      app: euw1-apm
      apm-infra: euw1-apm
        app: euw1-apm
        apm-infra: euw1-apm
      hostNetwork: true # NOTE: Agent is configured to have same IP as the host/node
      dnsPolicy: ClusterFirstWithHostNet
        - name: euw1-apm
          image: ""
          imagePullPolicy: IfNotPresent
            - "--strict.perms=false"
            - "-e"
            - "-E"
            - "output.elasticsearch.hosts=[\"http://replacing-me:9200\"]"
#                - "output.elasticsearch.hosts=[\"http://euw1-es-http:9200\"]"
#                - "-E"
#                - "output.elasticsearch.index=\"apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}\""
#                - "-E"
#                - "\"apm-server\""
#                - "-E"
#                - "setup.template.pattern=\"apm-%{[observer.version]}-%{[processor.event]}-*\""
            - containerPort: 8200
              protocol: TCP
              path: /
              port: 8200
            initialDelaySeconds: 60
              path: /
              port: 8200
            initialDelaySeconds: 60

Added more log when starting APM:

2020-01-21T04:05:49.232Z INFO [request] middleware/log_middleware.go:76 request ok {"request_id": "3f896fbd-aa69-4969-9e55-466722d3a023", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "", "user-agent": "kube-probe/1.14+", "response_code": 200}

2020-01-21T04:05:56.957Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://euw1-es-http.logging.svc.cluster.local:9200)): Connection marked as failed because the onConnect callback failed: resource 'apm-7.5.1-span' exists, but it is not an alias

2020-01-21T04:05:56.957Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://euw1-es-http.logging.svc.cluster.local:9200)) with 153 reconnect attempt(s)


Hi @Huy_Ngo, looking at your config file I wonder if you had run the APM Server with a custom output.elasticsearch.index configuration before, which would set up indices without ILM.

In 7.5. indices are by default setup using ILM, the according setting is apm-server.ilm.enabled: auto. If however a custom output.elasticsearch.index is configured, ILM is disabled by default. The config file you show suggests that you might have had APM Server running with some custom indices. In such a case you would need to set apm-server.ilm.setup.overwrite: true to allow APM Server to take care of setting up indices with ILM.

However, since in your case indices already exist that have the same name as the expected aliases, APM Server will not be able to overwrite those. Is it feasible for you to delete the troubling indices and restart APM Server with the default configuration for indices and ILM?

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.