Hi, I was tasked to upgrade elastic, kibana, logstash, and APM Server from 7.17 to 8.9, when I did the upgrade, old metrics from APM do not show in kibana APM section, only the new ones, so we restore the snapshot of the old stack.
We are using java agent and RUM
the APM server send data to logstash and then to elastic.
What could be wrong?
I also have this problem. Upon investigation, it seems that the 8.9.1 is using data stream while the old 7.x.x version are not using the new fields. On the 8.9.1, there is a new field metricset.samples.name regarding jvm, transactions, etc and metricset.samples.value. The APM Dashboard is still using the old 7.x.x settings.
The only solution we found was still use the 7.17.13 apm-server. The problem was the 8.x and might be the incorrect mapping.
This is the problem for us. It looks like a workaround maybe for us to create the ingest pipeline and copy the event.duration to transaction.duration.us
It's crucial to ensure compatibility between your Java agent, RUM, APM Server, and Elasticsearch versions when upgrading. Investigate version compatibility and configuration settings to resolve the issue and retain access to historical APM metrics in Kibana. Seek support or documentation from Elastic for detailed guidance. Best of luck with your upgrade! AC Football Cases.
Hi @ElasticLiver and @adsandie ,
I've managed to reproduce the issue where incomplete data is shown on APM UI after upgrade. Please be assured that there is no data loss as this is just a UI issue. This happens when the APM UI queries for a time range that covers the upgrade window, when both old and new data exists.
The root cause is that there are a lot of performance improvements in 8.7 apm-server (see changelog), with a major theme around "service metrics", which are pre-aggregated metrics to speed up APM UI (see blog post). An unfortunate side effect of introducing these new data streams is that when data (new data ingested since 8.9 upgrade) in the new pre-aggregated data streams exists, the UI returns immediately and do not fallback to query the old data streams (where your 7.17 data are). When the query only covers time before upgrade, because there is no data in new data streams, then the old data will show up.
The above implication will be documented as part of [APM] Document the UI query strategies for fetching data · Issue #155066 · elastic/kibana · GitHub
A workaround to potentially fix the UI weirdness is to disable service metrics under APM settings in Kibana. But there may be some graphs that are not affected by this feature toggle.
To reiterate, there is no data loss and as long as you don't query across the upgrade time, the UI should perform as expected. This also means that it will be lesser of an issue the longer you use the newer version.
I resolved this issue by creating the ingest pipeline and copying the value from event.duration to transaction.duration.us and span.duration.us and using 8.8.2 version. The 8.9.1 is another issue and needed time for us to troubleshoot further. RIght now, elasticsearch, kibana, logstash, beats are using 8.9.1 and apm-server is using 8.8.2. This is an upgrade from 7.17.x to 8.x
I was thinking to create a dashboard with some metrics of APM, So if the client want to view the old data has to visit the dashboard, and to view the new data has to use the the APM in the observability section, is this possible?
Hi @ElasticLiver,
It would require discipline in using date range filters for the dashboards and the APM UI, as there are no means to restrict the time range per page. You can share the links to the Dashboards with a specified time range though, and warn the clients that they won't see some data if they change the time range filters.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
