APM server not creating separate indices

Elastic Observability APM
1

Kibana version: 7.4.0

Elasticsearch version: 7.4.0

APM Server version: 7.4.0

Upgraded from other version: From 6.7.1

I upgraded elastic stack and APM server to 7.4.0. I installed APM server from deb package. Earlier SSL was enabled for output.elasticsearch in apm-server.yml but now I have removed it. Other settings are same. I am facing below three problems:

  1. Separate indices are not created for different processor.event. I see that only one index is created per day, for example today's index is apm-7.4.0-2019.12.17. I am using default setting for example
    index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
    when.contains:
    processor.event: "error"

  2. Whenever I try to see results in Kibana discover for index pattern apm-*, I get shard failed issue and the response is No field found for [context.service.name] in mapping with types []. I can see that the fields.yml file is different in 7.4.0 than 6.7.1 and there is no field for context. Should I copy the old fields.yml file and restart APM server?

  3. When I start APM server using systemctl start apm-server, logs are not being created in /var/log/apm-server/ but I can see logs in journalctl. When I start APM server using /usr/share/apm-server/bin/apm-server -c /etc/apm-server/apm-server.yml --path.home /usr/share/apm-server --path.config /etc/apm-server --path.data /var/lib/ap -server --path.logs /var/log/apm-server manually, I can see logs in /var/log/apm-server/. How can I configure APM server to send logs to default location when using systemctl command?

2

Hi @sharry007, sorry to hear you experience some issues on upgrading.

7.4.0 by default uses ILM for index management. I suggest you remove all output.elasticsearch.index and output.elasticsearch.indices.* configurations and leverage ILM, if you do not rely on customized indices. It sounds like the index template and Kibana index pattern might not have been created properly. For this I suggest you run apm-server setup to make sure indices, templates and patterns are properly setup.

You can find more information about default directories and how to configure APM Server when running with systemd.

3

Hey @simitt, thanks for replying.

I removed all output.elasticsearch.indices.* and tried to create custom lifecycle policy using apm-server.ilm setting. New indices are created but their naming is like apm-7.4.0-transaction-000001. How can I configure to create daily indices with date included in the index name?

Even after running apm-server setup, I don't see any field for context. I have a scripted field which queries for [context.service.name] but it is giving error No field found for [context.service.name] in mapping with types []. Earlier it used to work in APM-6.7.1.

And I wanted to use custom lifecycle policy and added the configuration in apm-server.yml but new indices were created with default ILM policy. I am attaching the configuration file for reference. Please help.

4

Looking at the attached apm-server.yml file you need to move apm-server.ilm.* configurations outside of output.elasticsearch.*, as apm-server.* is a top level configuration option.

When using ILM with APM Server you cannot add a date to the index, as index names are not configurable for ILM.
We previously created daily indices containing the date in the index name by default to avoid having one ever growing index. However, when enabling ILM, the ILM policy allows to define when an index should be rolled over to avoid ending up with huge indices. With ILM there is therefore no need for adding a date pattern to the index name anymore. May I ask for your use case of including a date in the index?

Regarding context.service.name, starting with 7.0 we introduced some some changes related to naming of APM data in Elasticsearch, as we are aiming to become Elastic Common Schema (ECS) compliant. The service name can be found under service.name now instead of context.service.name.

Hope this helps, let me know if there are any other questions.

5

Thanks for clearing this out.

I have been using date formatted index names for quite some time and I wish to continue using the same. But old configurations don't seem to work in my upgraded APM/ELK cluster. Can you suggest how can I get it to work like before? I am not too keen on using ILM for APM server so you can leave that out. And I also want to have separate indices for separate processor.event.

This is how my output.elasticsearch was in old configuration:

output.elasticsearch:
  hosts: ["<url>:9200"]
  protocol: "http"
  username: "user"
  password: "pass"
  indices:
    - index: "apm-%{[beat.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"
    - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"
    - index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"
    - index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"
    - index: "apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"
    - index: "apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"
  ssl.enabled: false

Everything else is same.

6

For 7.4 if you do not want to use ILM but it was already setup then you should be fine with changing your APM Server configuration to the following:

apm-server.ilm.enabled: false
setup.template.enabled: true
setup.template.overwrite: true
output.elasticsearch:
  hosts: ["<url>:9200"]
  protocol: "http"
  username: "user"
  password: "pass"
  index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"
  
    - index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"
  
    - index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"
  
    - index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"
  
    - index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"
  
    - index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"

Note that beat.version changed to observer.version from 7.0 on.

The indices are created as soon as data are ingested.

7

Thanks @simitt,

This helped me to get it working like before. New indices are created with date appended in the name.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.