How can I create a APM Template with timestamps and the 0000n (for ILM)

Elastic Observability APM
1

Hello to the community,
I am trying to setup APM with an ILM that I configured manually. If I use the default index available in the output.Elasticsearch, I get indexes similar to "apm-7.16.3-error-000001". I now need to add a timestamp to my indexes so that I can know at which date they were created without having to go thru the details of each index individually.

I have referred to the the below docs and pages and tried to get to the mentioned config :

apm-server:
  host: "0.0.0.0:8200"
  rum.enabled: "true"
  ilm:
    enabled: "true"
    setup:
      enabled: "true"
      overwrite: "true"
      require_policy: "false"
      mapping:
        - event_type: "error"
          policy_name: "My-kibana-ILM-1"
          index_suffix: ""
        - event_type: "span"
          policy_name: "My-kibana-ILM-1"
          index_suffix: ""
        - event_type: "transaction"
          policy_name: "My-kibana-ILM-1"
          index_suffix: ""
        - event_type: "metric"
          policy_name: "My-kibana-ILM-1"
          index_suffix: ""
setup:
  template:
    enabled: "true"
    overwrite: "true"
    name: "apm-%{[observer.version]}"
    pattern: "apm-%{[observer.version]}*"
output:
  elasticsearch:
    enabled: "true"
    protocol: "http"
    hosts: ["loggingcluster-master:9200"]
    index: "apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"
    indices:
     - index: "apm-%{[observer.version]}-sourcemap-%{+yyyy.MM.dd}"
       when.contains:
         processor.event: "sourcemap"
     - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
       when.contains:
         processor.event: "error"
     - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
       when.contains:
         processor.event: "transaction"
     - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
       when.contains:
         processor.event: "span"
     - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
       when.contains:
         processor.event: "metric"
     - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
       when.contains:
         processor.event: "onboarding"

I need my indexes to be created in this format : "apm-7.16.3-error-2022-04-13-000001" (need the date to change dynamically based on each day)

Currently with the above config, I am only able to achieve this :

image
image526×688 75.7 KB

I was able to configure my logstash (for reference, not to be confused with the APM settings)
image

Currently working on apm-server : version: 7.16.3

2

Hi @jehanjazz-1 ,

If the date suffix is the only part you want to change, you could still use the default ILM policy and only configure the index_suffix; something similar to

apm-server:
  ilm:
    enabled: "true"
    setup:
      enabled: true
      mapping:
        - event_type: "error"
          policy_name: "apm-rollover-30-days"
          index_suffix: "%{+yyyy-MM-dd}"
        - event_type: "span"
          policy_name: "apm-rollover-30-days"
          index_suffix: "%{+yyyy-MM-dd}"
...

would lead to indices such as apm-7.16.3-error-2022-04-20-000001 and apm-7.16.3-span-2022-04-20-000001 .

3

Hello,

Thank you for the suggestion. I will try to apply this in my config and let you know if all goes well.

Cheers !!

4

Hello @simitt

I tried the configuration, the Indices created were as per the way I needed them to be

image
image743×503 30.1 KB

Although now the Index template itself has a Date tag in it :

image
image878×668 51.4 KB

Is there any way to get the index template without the Date Tag? but get the indexes created with the Date and 0000n format?
Eg.
Index Template :
Name : apm-7.16.3-error
Pattern : apm-7.16.3-error*

Indices created by the above template :
Index name : apm-7.16.3-error-2022-04-23-000001
ILM : Same from my first config file.

For reference to get the output shown in the images I used this config :

apm-server:
  host: "0.0.0.0:8200"
  rum.enabled: "true"
  ilm:
    enabled: "true"
    setup:
      enabled: "true"
      overwrite: "true"
      require_policy: "false"
      mapping:
        - event_type: "error"
          policy_name: "My-kibana-logs-1"
          index_suffix: "%{+yyyy-MM-dd}"
        - event_type: "span"
          policy_name: "My-kibana-logs-1"
          index_suffix: "%{+yyyy-MM-dd}"
        - event_type: "transaction"
          policy_name: "My-kibana-logs-1"
          index_suffix: "%{+yyyy-MM-dd}"
        - event_type: "metric"
          policy_name: "My-kibana-logs-1"
          index_suffix: "%{+yyyy-MM-dd}"
queue: {}
output:
  elasticsearch:
    protocol: "http"
    hosts: ["loggingcluster-master:9200"]
5

No that is not possible, the index_suffix is also included into the template. Could you share your concerns and use cases for not including it?

6

Hello,

I want to create or setup the APM similar to my logstash setup.

Under Logstash the Index Template created is as such :
Name : logstash
Pattern : logstash*
Indices created : logstash-2022.04.18-000006

output {
          elasticsearch {
            hosts => ["http://loggingcluster-master:9200"]
            index => "logstash-%{+yyyy.MM.dd}"
          ilm_pattern => "{now/d}-000001"
          ilm_policy => "My-kibana-logs" 
          }
        }

Similar to the above setup and output I wanted to setup APM as well, so that I can manage the APM indices based on the date and time stamp, and incase any manual action is needed, searching via date in the UI becomes much simpler than searching via Created timestamp.

I understand that my question or expectation maybe incorrect, and if so I'll setup APM without the DD-MM-YYYY in the index then.

7

As showed above, you can use the index_suffix to configure a date if prefered, but it will also be part of the template. Generally for the ILM configuration and rollover the date part is not necessary though.

8

Thank you for your inputs.

I'll configure the index without the Date suffix as of now then, since the ILM works even without them.