APM The selected trace cannot be found after 7.17.1 -> 8.1 upgrade

[edwardcccc (Chen Chen)]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

Kibana version:
Docker 8.1.0

Elasticsearch version:
Docker 8.1.0

APM Server version:
Elastic Agent Fleet Server 8.1.0

APM Agent language and version:
Python Flask 6.8.1

Browser version:
Chrome 99

I have an existing installation of
Docker
Elasticsearch 7.17.1
Kibana 7.17.1
APM Server 7.17.1

I upgraded to 8.1.0 using
Docker
Elasticsearch 8.1.0 (docker.elastic.co/elasticsearch/elasticsearch:8.1.0)
Kibana 8.1.0 (docker.elastic.co/kibana/kibana:8.1.0)
APM Server 8.1.0 (docker.elastic.co/apm/apm-server:8.1.0)
Then I changed to use elastic agent with fleet server to swap out the above container
docker.elastic.co/beats/elastic-agent:8.1.0

apm events goes to Elasticsearch and shows up on the dashboard no problem

However, I cannot see the trace

I am able to see trace in the document

I can still see trace from earlier times which is ingested in 7.17.1

clicking on these two links doesnot show anything

I have tried to uninstall APM integration, remove assets using this button and reinstall apm integration, still has the problem

[swchandu (chandrakanth)]

@edwardcccc

install APM integration, remove assets using this button and reinstall apm integration, still has the problem

I was also facing the same issue after 8.0 upgrade. I have noticed a mapping error in the fleet-apm server logs. so, deleted apm related indexes and also datastreams(doubting this too). Re-integrated apm and able to get the traces.
I haven't done in-depth analysis as env is dev. Thanks for posting it.

[stuart.nelson]

Can you share your index templates, and check the server logs to see if there's a mapping error, like what @swchandu saw when they upgraded?