APM Server - not all variables are working for Indicies templates

Elastic Observability APM
1

Hello!

I have APM Server with default config and all is working fine. Now I want to split ES indices from APM by project or hostname ore something else and get a problem. I am using this part of config for this:

`
output.elasticsearch:
...

index: "apm-%{[observer.version]}-ident-%{+yyyy.MM.dd}"
indices:
- index: "apm-%{[observer.version]}-sourcemap"
when.contains:
processor.event: "sourcemap"

...

- index: "apm-%{[observer.version]}-span-%{[source.ip]}-int-%{+yyyy.MM.dd}"

`

And my issues is - all "variables", like a %{[source.ip]} is empty for me :frowning:
When I tried, for example, %{[observer.hostname]} - I got a new index with hostname in the name.

I tried at least all variables from this list - https://www.elastic.co/guide/en/apm/server/master/exported-fields-apm.html#_process

but all of them are empty (or not defined).

How I turn on them? Or maybe there is some other way to get hostname or application name from receiving data to add it to indice name?

2

Not all fields are set for all events. Specifically, source.ip is only set for some transaction docs - and no spans at all, currently.

How I turn on them? Or maybe there is some other way to get hostname or application name from receiving data to add it to indice name?

You can use host.name for the hostname of the instrumented service, or service.name for the service name.

3

@axw should I use it like:

 index: "apm-%{[observer.version]}-ident-%{+yyyy.MM.dd}"
  indices:
- index: "apm-%{[observer.version]}-sourcemap"
  when.contains:
    processor.event: "sourcemap"

- index: "apm-%{[observer.version]}-error-%{[host.name]}-%{[service.name]}-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "error"

- index: "apm-%{[observer.version]}-%{[host.name]}-%{[service.name]}-transaction-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "transaction"

- index: "apm-%{[observer.version]}-span-%{[host.name]}-%{[service.name]}-int-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "span"

- index: "apm-%{[observer.version]}-metric-%{[host.name]}-%{[service.name]}-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "metric"

- index: "apm-%{[observer.version]}-onboarding-%{[host.name]}-%{[service.name]}-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "onboarding"

?

If so - it does not work :frowning:

I have indexes with names like apm-7.6.1-transaction-2020.08.12 - so, looks like host.name and service.name are blank too for me :frowning:

4

@TheVintik that looks right, except for the onboarding one: those docs do not have a service.name. Can you please share your complete apm-server.yml config? (With any passwords redacted of course.)

5

Hello,

Additionally to what @axw asked, might I ask what agent are you using (language and version)?

6

@axw I am using Ruby on Rails application with elastic-apm (latest version). There is no any additional settings from Rails side except secret key to send data to APM.

7

@axw

Here is my config:

[centos@XXX ~]$ sudo cat /etc/apm-server/apm-server.yml | grep -v '#'

apm-server:
  host: "XXXX:8200"
  secret_token: XXXXX

  kibana:
    enabled: true
    host: "XXXXX:5601"
    protocol: "https"


setup.template.name: "apm-%{[observer.version]}-custom"
setup.template.pattern: "apm-%{[observer.version]}-*"
setup.template.overwrite: true


output.elasticsearch:
  hosts: ["XXXX:9200"]

  enabled: true

  index: "apm-%{[observer.version]}-ident-%{+yyyy.MM.dd}"
  indices:
    - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"
  
    - index: "apm-%{[observer.version]}-error-%{[host.name]}-%{[service.name]}-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"
  
    - index: "apm-%{[observer.version]}-%{[host.name]}-%{[service.name]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"
  
    - index: "apm-%{[observer.version]}-span-%{[host.name]}-%{[service.name]}-int-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"
  
    - index: "apm-%{[observer.version]}-metric-%{[host.name]}-%{[service.name]}-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"
  
    - index: "apm-%{[observer.version]}-onboarding-%{[host.name]}-%{[service.name]}-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"
8

that looks right, except for the onboarding one: those docs do not have a service.name .

The main goal for me - I need to split indices with APM metric data, and, as I know - they are located in span indexes (these indexes most huge as I can see)

9

The config looks fine. I copy & pasted it, and it worked when I ran it in Docker, placing the config file in /usr/share/apm-server.

GET /_cat/indices/apm-*

yellow open apm-7.6.1-goat-foo-transaction-2020.08.18 sMqPHbvDRY2PLKk2oRyAbA 1 1 1 0 230b 230b

(my host.name is "goat", and service.name is "foo")

Can you please share the apm-server log?

10

@axw looks like something was wrong with my stack of ES + Kibana + APM, because after reinstall all of this - I get necessary indices splitted by project, so, the config is really correct.

Thank you for your help!

11

Great, glad to hear you sorted it out!