# Traces-apm-default not create as data stream after migrate to APM integration **URL:** \ **Category:** APM\ **Tags:** ilm-index-lifecycle-management, server\ **Created:** [April 15, 2022, 4:06pm UTC](https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511 "2022-04-15T16:06:24Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![amelieBoond](https://avatars.discourse-cdn.com/v4/letter/a/6f9a4e/32.png) [@amelieBoond](https://discuss.elastic.co/u/amelieBoond)\ **Post date:** [April 15, 2022, 4:06pm UTC](https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511/1 "2022-04-15T16:06:24Z")
Hello, We have a problem after upgrade elastic/apm from version 7 to 8 and migrating to APM integration. We followed the guide and didn't missed any step. After the upgrade, a new index appear : trace-apm-default but this is a simple index not a data stream, so there is no ilm on it Is it possible to migrate this index (which is very big, 1,7T 😕 ) to a data stream ? It's on production stack, so I'm a little afraid to take action on it 😕 **Kibana version** : v 8.1.0 **Elasticsearch version** : v 8.1.0 **APM Server version** : v 8.1.0 **APM Agent language** : php I tried to reproduce on another server, so install again apm-server, enable APM integration and this time the trace-apm-default is a data stream. Thanks for helping ---
**Author:** ![slhck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slhck/32/32578_2.png) [@slhck](https://discuss.elastic.co/u/slhck)\ **Post date:** [May 2, 2022, 9:32am UTC](https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511/2 "2022-05-02T09:32:35Z")
Did you figure this out? Same for me: I have a traces-apm-default index, no data stream. No ILM is configured, so this index keeps blowing up, eating up all disk space. I have an ELK stack that I have continuously updated from earlier ELK versions until the most recent one. Seems that some migration was botched here. The tutorial says that it should be a data stream: [Index lifecycle management | APM User Guide [8.1] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/ilm-how-to.html) There is a "traces-apm.traces-default\_policy" policy that was added, but it's not linked to any index. ---
**Author:** ![slhck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slhck/32/32578_2.png) [@slhck](https://discuss.elastic.co/u/slhck)\ **Post date:** [May 6, 2022, 10:41am UTC](https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511/3 "2022-05-06T10:41:59Z")
I've found a solution with help from the Elastic team: > > > \*\*APM Server version\*\* apm-server version 8.1.3 (amd64), libbeat 8.1.3 \[1c7bc9e0…bb5a3ab54cb67a881c49b4fda7a62241 built 2022-04-19 07:04:09 +0000 UTC\] > > \*\*Description of the problem including expected versus actual behavior\*\*: > > I have two ELK clusters that I have upgraded from 7.x to 8.x using the recommended migration guides. I have set up APM Server binary with a few agents sending data using Ruby and Python libraries. Both operate fine, except that the APM traces are now filling up the disk space. > > Here is the index: > > !\[image\](https://user-images.githubusercontent.com/582444/166216201-7179be74-3cdd-4e09-a388-77a8212c86f7.png) > > Here are the templates: > > !\[image\](https://user-images.githubusercontent.com/582444/166216267-88d9bbc8-c901-4f6a-b557-e4a9f6ddd205.png) > > As you can see, the default policy for traces is not applied: > > !\[image\](https://user-images.githubusercontent.com/582444/166216336-6c19d909-1e15-4ba0-bd4a-3ee81070ad09.png) > > I am unable to create my own ILM policy with rollup, since the traces-apm-default index is just an index, not an index pattern with different underlying indices; just one huge index that eats up disk space. > > I have tried \[migrating to data stream\](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-migrate-to-data-stream.html#indices-migrate-to-data-stream-request) but it fails with: > > \`\`\` > ➜ curl -k -XPOST -u elastic:XXXXX https://localhost:9200/\_data\_stream/\_migrate/traces-apm-default > {"error":{"root\_cause":\[{"type":"illegal\_argument\_exception","reason":"alias \[traces-apm-default\] does not exist"}\],"type":"illegal\_argument\_exception","reason":"alias \[traces-apm-default\] does not exist"},"status":400}% > \`\`\` > > Also, using \`traces-apm\*\` does not work: > > \`\`\`json > {"error":{"root\_cause":\[{"type":"security\_exception","reason":"action \[indices:admin/data\_stream/migrate\] is unauthorized for user \[elastic\] with roles \[superuser\], this action is granted by the index privileges \[manage,all\]"}\],"type":"security\_exception","reason":"action \[indices:admin/data\_stream/migrate\] is unauthorized for user \[elastic\] with roles \[superuser\], this action is granted by the index privileges \[manage,all\]","caused\_by":{"type":"illegal\_argument\_exception","reason":"the action indices:admin/data\_stream/migrate does not support wildcards; the provided index expression(s) \[traces-apm\*\] are not allowed"}},"status":403} > \`\`\` > > Here is another user experiencing exactly the same issue: https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511 > > \*\*Steps to reproduce\*\*: > > \- Have a cluster at latest 7.x > \- Upgrade to 8.x > > \*\*Provide logs (if relevant)\*\*: > > I have no migration logs available unfortunately. This is apparently a bug with APM, and you need to ensure that you turn off APM server, then remove the APM integration entirely, delete the wrong index, and install the integration again. Only then turn on APM server. See the GitHub thread for more info. ---
**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\ **Post date:** [May 6, 2022, 11:24am UTC](https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511/4 "2022-05-06T11:24:31Z")
Hi @amelieBoond , apologies for coming back to this topic that late. With the help of @slhck we were able to [identify a bug in the APM Server](https://github.com/elastic/apm-server/issues/8018#issuecomment-1119443116): when the APM Integration hasn't been installed before the APM Server is stopped or restarted, it will drop any queued events to Elasticsearch and an index is created because of the missing index templates. Unfortunately the easiest way to solve this issue would also mean loosing data: (1) Stop APM Server (2) [Install the APM Integration via Fleet UI](https://www.elastic.co/guide/en/apm/guide/current/apm-quick-start.html#add-apm-integration) (3) Delete the `traces-apm-default` index (4) Start the APM Server If you are keen on keeping the existing data, after step (1)+(2) you could try to reindex the data from the index to a data stream which matches the `traces-apm-*` index pattern, for example ```auto POST _reindex { "source": { "index": "traces-apm-default" }, "dest": { "index": "traces-apm-backup", "op_type": "create" } } ``` This would probably mean a longer interruption on the cluster though as you would have to disable APM Server for the reindexing time. I'm also not certain how managable the reindexing is for such a large index. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 27, 2022, 7:24am UTC](https://discuss.elastic.co/t/traces-apm-default-not-create-as-data-stream-after-migrate-to-apm-integration/302511/5 "2022-05-27T07:24:47Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.