# APM server ILM indices not getting any documents - only apm-%{version}

**URL:** <https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196>\
**Category:** APM\
**Tags:** server\
**Created:** [April 28, 2020, 2:22pm UTC](https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196 "2020-04-28T14:22:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cotjoey](https://avatars.discourse-cdn.com/v4/letter/c/76d3ee/32.png) [@cotjoey](https://discuss.elastic.co/u/cotjoey)\
**Post date:** [April 28, 2020, 2:22pm UTC](https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196/1 "2020-04-28T14:22:07Z")

</div>

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂

**Kibana version** : 7.6.0

**Elasticsearch version** : 7.6.0

**APM Server version** : 7.6.0

**APM Agent language and version** : 1.12

**Browser version** : Edge

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: RPM package (yum on Red Hat)

**Fresh install or upgraded from other version?** Fresh install

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.  
We are using Logstash with a custom config file with snippet that sets the index:

> ```
> } else if [@metadata][beat] == "apm" {
> mutate {
> add_field => {
> "[@metadata][index_prefix]" => "%{[@metadata][beat]}-%{[@metadata][version]}"
> "[@metadata][doc_id]" => "%{[@metadata][uuid]}"
> }
> }
> 
> ```

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:  
I want to use ILM indices for APM, but I don't get any data in the indices that automatically get created by the setup command.

After I run setup, I see the following indices created (apm-server.yml: ilm.enabled: "true":  
apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

apm-7.6.0-profile-000001  
apm-7.6.0-metric-000001  
apm-7.6.0-error-000001  
apm-7.6.0-transaction-000001  
apm-7.6.0-span-000001

Those 5 automatically-created indices don't get any data from Logstash. They stay with a size of 466b/566b and never get any documents.

I am not sure if I am missing something to the setup.

All the APM events get bundled under one index called "apm-7.6.0" which doesn't have a lifecycle policy.

**Steps to reproduce** :

1. on apm-server run setup command above;
2. start apm-server
3. look at indices in Kibana

**Errors in browser console (if relevant)**: No

**Provide logs and/or server output (if relevant)**: N/A for now

---

<div class="post-metadata">

**Author:** ![bmorelli25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bmorelli25/32/38738_2.png) [@bmorelli25](https://discuss.elastic.co/u/bmorelli25)\
**Post date:** [April 28, 2020, 2:55pm UTC](https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196/2 "2020-04-28T14:55:31Z")

</div>

Hi Joey,

I'm sorry you're having trouble. I assume you're using Logstash's Elasticsearch output plugin? What does that Logstash output look like? Did you follow the pattern shown in the [Logstash and ILM](https://www.elastic.co/guide/en/apm/server/current/logstash-output.html#_logstash_and_ilm) documentation? It looks like you're outputting `"[@metadata][index_prefix]"` which needs to also include `-%{[processor][event]}` (this is where the `error`, `span`, `transaction`, etc. comes from), which I don't see in your mutate filter.

---

<div class="post-metadata">

**Author:** ![cotjoey](https://avatars.discourse-cdn.com/v4/letter/c/76d3ee/32.png) [@cotjoey](https://discuss.elastic.co/u/cotjoey)\
**Post date:** [April 28, 2020, 3:04pm UTC](https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196/3 "2020-04-28T15:04:56Z")

</div>

Oh my. Yes, you are right! I added snippets of IF/ELSE statements to Logstash as such:

```
if [processor][event] == "profile" {
        mutate {
          add_field => {
            "[@metadata][index_prefix]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}"
            "[@metadata][doc_id]" => "%{[@metadata][uuid]}"
          }
        }
      }

```

... and I started seeing events being put into the proper indices.

Thank you for pointing me out to the -%{[processor][event]} variable.

---

<div class="post-metadata">

**Author:** ![bmorelli25](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bmorelli25/32/38738_2.png) [@bmorelli25](https://discuss.elastic.co/u/bmorelli25)\
**Post date:** [April 28, 2020, 3:38pm UTC](https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196/4 "2020-04-28T15:38:08Z")

</div>

Awesome! I'm glad to hear you got things working. As a follow-up, I've opened an [issue](https://github.com/elastic/apm-server/issues/3705) to better explain the `processor.event` field in the Logstash output documentation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 19, 2020, 11:38am UTC](https://discuss.elastic.co/t/apm-server-ilm-indices-not-getting-any-documents-only-apm-version/230196/5 "2020-05-19T11:38:11Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.