Using legacy APM setup after upgrading Elasticsearch to 8.X

Elastic Observability APM
1

Hi,

I have a self-managed setup on v7.9.2 that ooks like this: Custom APM agents (hundreds) --> load balancer --> APM servers --> Kafka for MQ --> Logstash servers --> Elasticsearch servers

I upgraded a development stack to 8.6.2. I tinkered with it for a couple of hours and got it working. I see that data is indexed to 'apm-server-8.6.2-2023.04.04' but I can't see new data in the 'APM' menu. After a while, I checked and saw that the mapping was wrong (not applied). Per the documentation, I installed the integration. I'm not sure how I can set the mapping up on a 8.X stack, I don't think copying the old template would be sufficient. Could someone help with this issue?

Thanks!

2

No takers?

3

Hi @YvorL , apologies for the late reply, this somehow fell through the cracks.
Apm server changed to write to datastreams in 8.0+, of the format {data_stream.type}-{data_stream.dataset}-{data_stream.namespace}. For example this leads to datastreams such as traces-apm-default. If installing the apm integration, then the mappings and ILM policies are set up in accordance with this. The namespace part is where you can customize the datastreams. Keeping custom logic that sends data to apm-* indices leads to default templates not being applied. In such a case, you would need to set up templates yourself.

When using apm server standalone (not managed by Elastic Agent), sending data to Kafka/Logstash is still supported in 8.6. But you probably want to update your logstash configuration to send the data to datastreams in alignment with the default setup. Then installing the apm integration is all you have to do for templates & ILM policies to get applied.

4

Thanks! I can't test this, I deleted the idling dev server today and moved on to another project in the meantime.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.