Index name must be lowercase after upgrade to 7.9.2

Elastic Observability APM
1

Hello,

Does anybody know - is it possible to transform index name to lowercase in APM config file?

Started to getting warning after apm upgrade to latest version:

WARN	[elasticsearch]	elasticsearch/client.go:407	Cannot index event publisher.Event{Content:beat.Event ... n":"Invalid index name [apm-7.9.2-span-SomeNewIndex-2020.10.08], must be lowercase","index_

I am using this config for custom indexes and it was worked for me on 7.7.1:

index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"
    - index: "apm-%{[observer.version]}-error-%{[service.name]}-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

So, looks like I need to find a way to transform %{[service.name]} to lowercase. Is it possible?

2

May be an ingest pipeline to update document metadata like _index followed by a lower case processor

3

will check but looks like it is ES transformation, but my issue is in APM, not sure about this...

4

You can simply make sure all services instrumented by Elastic APM agents have a lower-case name

5

Elasticsearch is limiting index names to lowercase characters only, that did not change between 7.7 and 7.9 though, refer to ES index name restrictions (7.7).

You can manually configure the service name for every APM agent; therefore as @ylasri pointed out, you could ensure that your service names fulfill all ES index name requirements when using them as part of ES index names.

6

that did not change between 7.7 and 7.9 though,

Interesing... Will double check, but today I had upgraded my ES and Kibana from 7.7 to 7.9 - and it is working as expected.

I have three APM Server instances with the same configs.

I stopped all apm server services and upgraded apm server on the first instance and run it. After that I saw this warning and I stopped getting any data in APM (there were no any data in Kibana/APM). I stopped apm server 7.9 and run apm in other 2 instances with APM 7.7 - and there is not any errors and I started getting data.

So, for now, I am trying to figure out - what is wrong with 7.9. All instances have the same configs.

7

Thanks for the details @TheVintik,
I was able to reproduce - the service.name ended up as a lower case index name in 7.7, but does not anymore since 7.8. I am going to look into that in more detail and will come back to you.

In the meantime, you could make use of configuring the service name in the agents (as suggested above) as a workaround.

8

Thank you, will wait for possible solutions from you. It is hard to setup service name in the agents for now :frowning:

9

Thanks again for reporting; I identified the issue, you can follow along at https://github.com/elastic/apm-server/issues/4294

10

The fix has been merged and will be released with the next 7.9.x release.

1 Like
11

awesome, thank you!

12

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.