Downgrading the standalone agent causes the services and transactions to disapper from Kibana

Elastic Observability APM
1

Kibana version: 7.8.1

Elasticsearch version: 7.8.1

APM Server version: 7.8.1

APM Agent language and version: Java

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
We are having Weblogic (12.2.1) running in the production environment. We attached the stanalone apm jar to the PID. (Refer https://discuss.elastic.co/c/observability/apm/58)

We downgraded the APM version from 1.18 (standalone) to 1.17 (standalone).
When we accessed the Kibana APM services tab, the previously discovered services/traces etc dont' appear and only the service corresponding to 1.17 appears.

This is the exception while attaching the agent,
2020-09-15 13:00:54,748 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.17.0 as weblogic-PID-17 on Java 1.8.0_265 (Private Build) Linux 4.4.0-1113-aws
2020-09-15 13:00:54,748 [Attach Listener] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages
2020-09-15 13:00:54,831 [Attach Listener] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-09-15 13:00:54,841 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2020-06-14T17:10:16Z", "build_sha": "06c58bf4e5b675d04314bf44961ffd6b0e13f544", "version": "7.8.0"}
2020-09-15 13:00:54,918 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Remote configuration is not available. Check the connection between APM Server and Kibana.

Request your help in this.

  1. On another note, is it possible to have a combination of java agents while monitoring applications - say Jboss monitored with 1.18 (standalone) and weblogic with 1.15 (standalone) agents.

  2. When we use instrument as false , enable as false in elasticsearch.properties , the JVM related data is still coming to APM. Is that how the functionality is? Is that that only traces are not available.

Thanks again for all your help!

Regards,
Bhavani

2

Hello Team,

Please let me know. Do I need to delete all the indices/reinstall APM? How can the older data be retrieved?

Thanks and Regards,
Bhavani

3

Hi @Bhavani_Ananth,

There are quite a few things in your message, thus I'll start by trying to properly understand what could happen here.

  • you are using apm-server 7.8, with java agent 1.18.0, later downgraded to 1.17.0, did you changed anything else apart from the agent ?

  • is the elasticsearch.properties the file that you are using to configure the agent with the config_file option ? Or is it a file in your application classpath ?

The following message indicates that remote configuration is probably not enabled on your server, did it already happened with 1.17.0 agent or is it something new ?

Yes, you can use different agent versions for different JVMs, while we recommend to always use the latest our agent and servers remain compatible.

When you change the value in elasticsearch.properties, you should see a log message when configuration is refreshed.

4

Thanks so much Sylvain for your detailed response.

a. No , nothing was changed apart from downgrading the agent from 1.18 to 1.17. I think it caused some issues in mapping. I followed the steps https://www.elastic.co/guide/en/apm/server/current/common-problems.html#server-resource-exists-not-alias and was able to get the APM working.

a.1 The old data is still not visible though. Do we reindex the cloned index(apm-7.9.1-transaction.original) to the new index genereated by APM (apm-7.9.1-transaction-000001)

b. The elasticsearch.properties was used with config_file option. Since we were dealing directly with production systems, we needed a way to detach the standalone agent if it did not work. (we did not follow the recommended way - through startup script and server restart). Though we gave the right parameters for detachment, the JVM metrics for that service are still coming in the APM app. Transactions, traces, spans etc. are not visible.

Thanks and Regards,
Bhavani

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

6

Hi,

We have recently released the 1.18.1 version, could you try to update to this one ?
What was the initial motivation for using different versions, was there any blocking issue that required to have the version 1.17.0 in use instead of the 1.18.x ?

Also, setting instrument=false will only disable instrumentation, it does not disable metrics collection (which JVM dashboards use).