Traces-apm-default not create as data stream after migrate to APM integration

Elastic Observability APM
,
1

Hello,

We have a problem after upgrade elastic/apm from version 7 to 8 and migrating to APM integration.
We followed the guide and didn't missed any step.

After the upgrade, a new index appear : trace-apm-default but this is a simple index not a data stream, so there is no ilm on it

Is it possible to migrate this index (which is very big, 1,7T :confused: ) to a data stream ?

It's on production stack, so I'm a little afraid to take action on it :confused:

Kibana version: v 8.1.0

Elasticsearch version: v 8.1.0

APM Server version: v 8.1.0

APM Agent language: php

I tried to reproduce on another server, so install again apm-server, enable APM integration and this time the trace-apm-default is a data stream.

Thanks for helping

2

Did you figure this out?

Same for me: I have a traces-apm-default index, no data stream. No ILM is configured, so this index keeps blowing up, eating up all disk space. I have an ELK stack that I have continuously updated from earlier ELK versions until the most recent one. Seems that some migration was botched here.

The tutorial says that it should be a data stream: Index lifecycle management | APM User Guide [8.1] | Elastic

There is a "traces-apm.traces-default_policy" policy that was added, but it's not linked to any index.

3

I've found a solution with help from the Elastic team:

This is apparently a bug with APM, and you need to ensure that you turn off APM server, then remove the APM integration entirely, delete the wrong index, and install the integration again. Only then turn on APM server. See the GitHub thread for more info.

4

Hi @amelieBoond ,
apologies for coming back to this topic that late.

With the help of @slhck we were able to identify a bug in the APM Server: when the APM Integration hasn't been installed before the APM Server is stopped or restarted, it will drop any queued events to Elasticsearch and an index is created because of the missing index templates.

Unfortunately the easiest way to solve this issue would also mean loosing data:
(1) Stop APM Server
(2) Install the APM Integration via Fleet UI
(3) Delete the traces-apm-default index
(4) Start the APM Server

If you are keen on keeping the existing data, after step (1)+(2) you could try to reindex the data from the index to a data stream which matches the traces-apm-* index pattern, for example


POST _reindex
{
  "source": {
    "index": "traces-apm-default"
  },
  "dest": {
    "index": "traces-apm-backup",
    "op_type": "create"
  }
}

This would probably mean a longer interruption on the cluster though as you would have to disable APM Server for the reindexing time. I'm also not certain how managable the reindexing is for such a large index.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.