Storing APM data in custom/specific indices

Meta Elastic Elastic Tips and Common Fixes
1

If you'd like to split your APM indices by the service, you can use the advice from this topic.

You can change the index name by setting either output.elasticsearch.index or output.elasticsearch.indices. These configuration variables take an index name "format string", which can reference fields the events. For your use-case, you can include %{[service.name]} in the format string to create an index per service.

e.g.

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  indices:
   - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"

   - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"

   - index: "apm-%{[observer.version]}-%{[service.name]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

   - index: "apm-%{[observer.version]}-%{[service.name]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"

   - index: "apm-%{[observer.version]}-%{[service.name]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"

   - index: "apm-%{[observer.version]}-%{[service.name]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"

Note that the "onboarding" and "sourcemap" indices are not service-specific, hence we did not include %{[service.name]} in their names in this example.

2

Thanks for writing this! As a call out for people following this outline, from the associated doc, note the disclaimer that this turns off the default ILM policy:

Defining a custom index here will disable Customize index lifecycle management .

This implicates a user responsibility to setup customized, self-managed ILM policies if they chose to implement this sub-configuration.

(Not a blocker in any way, but a necessary setup expectation detail :slight_smile: )

3 Likes
4

Note this is applicable only on APM Server (legacy). For new versions and for APM Integration, the indices are actually data streams.