APM server ILM indices not getting any documents - only apm-%{version}

[cotjoey (Joey)]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

Kibana version: 7.6.0

Elasticsearch version: 7.6.0

APM Server version: 7.6.0

APM Agent language and version: 1.12

Browser version: Edge

Original install method (e.g. download page, yum, deb, from source, etc.) and version: RPM package (yum on Red Hat)

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
We are using Logstash with a custom config file with snippet that sets the index:

  } else if [@metadata][beat] == "apm" {
      mutate {
        add_field => {
          "[@metadata][index_prefix]" => "%{[@metadata][beat]}-%{[@metadata][version]}"
          "[@metadata][doc_id]" => "%{[@metadata][uuid]}"
        }
      }

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I want to use ILM indices for APM, but I don't get any data in the indices that automatically get created by the setup command.

After I run setup, I see the following indices created (apm-server.yml: ilm.enabled: "true":
apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

apm-7.6.0-profile-000001
apm-7.6.0-metric-000001
apm-7.6.0-error-000001
apm-7.6.0-transaction-000001
apm-7.6.0-span-000001

Those 5 automatically-created indices don't get any data from Logstash. They stay with a size of 466b/566b and never get any documents.

I am not sure if I am missing something to the setup.

All the APM events get bundled under one index called "apm-7.6.0" which doesn't have a lifecycle policy.

Steps to reproduce:

1. on apm-server run setup command above;
2. start apm-server
3. look at indices in Kibana

Errors in browser console (if relevant): No

Provide logs and/or server output (if relevant): N/A for now

[bmorelli25 (Brandon Morelli)]

Hi Joey,

I'm sorry you're having trouble. I assume you're using Logstash's Elasticsearch output plugin? What does that Logstash output look like? Did you follow the pattern shown in the Logstash and ILM documentation? It looks like you're outputting "[@metadata][index_prefix]" which needs to also include -%{[processor][event]} (this is where the error, span, transaction, etc. comes from), which I don't see in your mutate filter.

[cotjoey (Joey)]

Oh my. Yes, you are right! I added snippets of IF/ELSE statements to Logstash as such:

if [processor][event] == "profile" {
        mutate {
          add_field => {
            "[@metadata][index_prefix]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}"
            "[@metadata][doc_id]" => "%{[@metadata][uuid]}"
          }
        }
      }

... and I started seeing events being put into the proper indices.

Thank you for pointing me out to the -%{[processor][event]} variable.

[bmorelli25 (Brandon Morelli)]

Awesome! I'm glad to hear you got things working. As a follow-up, I've opened an issue to better explain the processor.event field in the Logstash output documentation.