Default APM Server docker did not create apm-7.5.1-transaction and apm-7.5.1-span with suffix number behind

[Huy_Ngo]

Kibana version: 7.5.1

Elasticsearch version: 7.5.1

APM Server version: 7.5.1

APM Agent language and version: go

Browser version: Version 79.0.3945.88 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker image apm-server:7.5.1

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup?
We're using fluent bit to ship log to Elasticsearch

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
After install apm-server and access to output Elasticsearch, checked the Indice I don't see the indices had been created with "apm-7.5.1-transaction-000001" and "apm-7.5.1-span-000001". That make the life cycle rotate did not work for two indices above. Error generated:
illegal_argument_exception: index.lifecycle.rollover_alias [apm-7.5.1-transaction] does not point to index [apm-7.5.1-transaction]

⎈ |PRODUCTION_USE1:logging)]$ curl   localhost:9200/_cat/indices?v | grep span
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Handling connection for 9200
100  4953  100  4953    0     0   3984      0  0:00:01  0:00:01 --:--:--  3987
green  open   apm-7.5.1-span                  FnJJgBSUTOKpmRf3mux1yg   1   1   43516421            0     25.8gb         12.9gb

Steps to reproduce:

1. Install apm as K8s DaemonSet
2. Verify that apm can output to ES successfully
3. Check the indices make sure ES has Indices with suffix apm-7.5.1-transaction-000001

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: euw1-apm
  namespace: logging
  labels:
    app: euw1-apm
    apm-infra: euw1-apm
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 100
  selector:
    matchLabels:
      app: euw1-apm
      apm-infra: euw1-apm
  template:
    metadata:
      labels:
        app: euw1-apm
        apm-infra: euw1-apm
    spec:
      hostNetwork: true # NOTE: Agent is configured to have same IP as the host/node
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: euw1-apm
          image: "docker.elastic.co/apm/apm-server:7.5.1"
          imagePullPolicy: IfNotPresent
          args:
            - "--strict.perms=false"
            - "-e"
            - "-E"
            - "output.elasticsearch.hosts=[\"http://replacing-me:9200\"]"
#                - "output.elasticsearch.hosts=[\"http://euw1-es-http:9200\"]"
#                - "-E"
#                - "output.elasticsearch.index=\"apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}\""
#                - "-E"
#                - "setup.template.name=\"apm-server\""
#                - "-E"
#                - "setup.template.pattern=\"apm-%{[observer.version]}-%{[processor.event]}-*\""
            
          ports:
            - containerPort: 8200
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: 8200
            initialDelaySeconds: 60
          readinessProbe:
            httpGet:
              path: /
              port: 8200
            initialDelaySeconds: 60

Added more log when starting APM:

2020-01-21T04:05:49.232Z INFO [request] middleware/log_middleware.go:76 request ok {"request_id": "3f896fbd-aa69-4969-9e55-466722d3a023", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "172.16.158.138", "user-agent": "kube-probe/1.14+", "response_code": 200}

2020-01-21T04:05:56.957Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://euw1-es-http.logging.svc.cluster.local:9200)): Connection marked as failed because the onConnect callback failed: resource 'apm-7.5.1-span' exists, but it is not an alias

2020-01-21T04:05:56.957Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://euw1-es-http.logging.svc.cluster.local:9200)) with 153 reconnect attempt(s)

Thanks,
Huy

[simitt (Silvia Mitter)]

Hi @Huy_Ngo, looking at your config file I wonder if you had run the APM Server with a custom output.elasticsearch.index configuration before, which would set up indices without ILM.

In 7.5. indices are by default setup using ILM, the according setting is apm-server.ilm.enabled: auto. If however a custom output.elasticsearch.index is configured, ILM is disabled by default. The config file you show suggests that you might have had APM Server running with some custom indices. In such a case you would need to set apm-server.ilm.setup.overwrite: true to allow APM Server to take care of setting up indices with ILM.

However, since in your case indices already exist that have the same name as the expected aliases, APM Server will not be able to overwrite those. Is it feasible for you to delete the troubling indices and restart APM Server with the default configuration for indices and ILM?