Error while fetching resource Kibana APM

[danksim (Daniel Sim)]

elasticsearch, logstash, kibana: 7.17.3
apm-server: 7.5.2

I am getting the following error when trying to access the APM tab on Kibana:

Error while fetching resource
search_phase_execution_exception: [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory. (500)

This seems similar to the question here but I don't have any of the setup process in apm-server.yml as I am writing directly to kafka (not elasticsearch) and ilm setup is all taken care by logstash (see below).

Background:
I am using apm-server to output to a kafka topic named apm-server and I have logstash consume said kafka topic and output to elasticsearch.

apm-server config:

    # ensure only kafka output
    output.file:
      enabled: false
    output.console:
      enabled: false
    output.kafka:
      hosts: ["kafka-bootstrap.kafka:9092"]
      topic: apm-server
      key: '%{[processor.event,service.name,kubernetes.pod.uid]:default}'
      partition.hash:
        hash:
        - processor.event
        - service.name
        - kubernetes.pod.uid
        random: true
      compression: lz4

output piece of logstash pipeline:

    output {
      elasticsearch {
        hosts => ["https://elastic-stack-coordinator:9200"]
        ssl => true
        keystore => <REDACTED>
        keystore_password => <REDACTED>
        truststore => <REDACTED>
        truststore_password => <REDACTED>
        manage_template => false
        user => apm_writer
        password => <REDACTED>
        index => "apm-index"
        ilm_rollover_alias => "apm-index"
        ilm_pattern => "000001"
        ilm_policy => "apm-ilm"
        ilm_enabled => true
      }
    }

so when my ELK cluster starts, I end up with an index named apm-index-000001 with ilm_rollover_alias apm-index.

This all works fine in dev (identical settings as test cluster) and prod clusters but does not in a third cluster I just setup.

[danksim (Daniel Sim)]

Ah my apologies. There is a kubernetes job that runs the following legacy apm-server setup command. This is a snippet from helmfile sync:

        args:
        - "setup"
        - "-e"
        - "--index-management"
        - "-E=setup.template.pattern='apm-*'"
        - "-E=setup.template.name='apm-ns'"
        - "-E=setup.template.overwrite='true'"
        - "-E=setup.template.settings={'index.max_docvalue_fields_search': '20000','index.number_of_shards': '6', 'index.refresh_interval': '60s','index.lifecycle.name': 'apm-ilm','index.lifecycle.rollover_alias': 'apm-index', 'index.translog.durability': 'async'}"
        - "-E=setup.template.append_fields=[{name:'labels.queue',type:'keyword'},{name:'labels.topic',type:'keyword'},{name:'http.request.headers',type:'object',dynamic:'true'},{name:'http.request.headers.Encrypted-User-Uuid',type:'keyword'},{name:'http.request.headers.X-Encrypted-User-Uuid',type:'keyword'},{name:'http.request.headers.X-App-Name',type:'keyword'}]"
        - "-E=output.elasticsearch.index='apm-%!{(MISSING)[kubernetes][namespace]}-%!{(MISSING)+MM.dd.YYYY}'"
        - "-E=output.elasticsearch.username=${ELK_USERNAME}"
        - "-E=output.elasticsearch.password=${ELK_PASSWORD}"
        - "-E=output.elasticsearch.ssl.certificate=/etc/elk/CA/tls.crt"
        - "-E=output.elasticsearch.ssl.key=/etc/elk/CA/tls.key"
        - "-E=output.elasticsearch.ssl.certificate_authorities=['/etc/elk/CA/ca.crt']"
        - "-E=output.elasticsearch.hosts=['https://elastic-stack-coordinator:9200']"

If I were to implement the following fix, how would I use the setup command?

It looks like index.indices but then it's a list with - index. Am I able to do -E=output.elasticsearch.index.indices.index=<some-index>?