APM stop working after upgrade from 7.17.6 --> 8.4.2

Elastic Observability APM
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version:
8.4.2

Elasticsearch version:
8.4.2

APM Server version:
8.4.2

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
yum from version 7.x or maybe 6.x then upgrade a lot of time without problems so far.

Fresh install or upgraded from other version?
a long history of upgrade last one is from 7.17.6 to 8.4.2

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

In the apm-server.yml I have:

  • queue.mem.events: 5120 <--- maybe defined long time ago because of some problem
  • hosts
  • username and password
  • indices:
    • index: "apm-%{[observer.version]}-sourcemap"
      when.contains:
      processor.event: "sourcemap"
      ...
      I have several index templates all built the same way with filter on processor.event

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
The main problem is that there's no more APM data indexed.
So far what I did: update to last 7.x branch everything was working I had APM data, then I upgrade Elastic, Kibana and APM to 8.x by updating the file in repos.d and run yum update.
After some setup for the TLS Elastic, Kibana and Filebeat were working.
For APM I was only able to see the old data before upgrading so I check the doc and follow the instructions by going to integrations and add APM, On the interface after click on "Check APM server status" it return:

No APM Server detected. Please make sure it is running and you have updated to 7.0 or higher.

So I run the usual instruction the common problems page (I already had problem when upgrading and so far I was always able to fix).

  1. first check that apm were correctly upgraded by running: apm-server version

apm-server version 8.4.2 (amd64), libbeat 8.4.2 [3b392bf54b718b214ac793a9a9720e8f0f5117dc built 2022-09-14 08:59:02 +0000 UTC]

  1. Checked that the service was running with "service apm-server status" and service was running.

  2. Go back to the interface and click on "Check APM server status" but without success.

  3. Run apm-server test config

Exiting: could not initialize the keystore: config file ("/var/lib/apm-server/apm-server.keystore") must be owned by the user identifier (uid=0) or root

  1. Update the ownership this way chown root:apm-server apm-server.keystore

  2. Run again apm-server test config and get same error for the apm-server.yml, fix it same way as above.

  3. Run again apm-server test config

Config OK

  1. Run apm-server test output

elasticsearch: http://localhost:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 127.0.0.1
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 8.4.2

  1. Go back to to Kibana and click again on "Check APM server status" but still same No server detected

  2. Restart APM server though that need to restart again after the chown changes and after that APM no more start, there's no error logs in apm-server-20220926.ndjson only log info.

  3. put back the original chown apm-server:apm-server on the files the service start and run but in Kibana it say that it can't connect and "apm-server test config" return ownership error.

Here I'm stuck without APM data, and no solution so far. One guess is that there's some change between old version that I did not fix correctly and now as it deprecated it no more work, but can't figure out what. Thank for any help or tips.
If I go to Obervability > Service I can still access my old data, i.e. before upgrade which somehow it's normal as I still have the index.

Steps to reproduce:

  1. Upgrade from 7.17.6 to 8.4.2
  2. Details are above
2

I reply to myself there's no problem, in upgrading I only mess the logs that was saying

accept4: too many open files;

After check on stackoverflow I found the solution like always, written in the doc:

So I follow the doc and after restart get this hint in the logs "service apm-server status -l"

please install the apm integration: Quick start with Elastic Cloud | APM User Guide [8.11] | Elastic"

Follow the doc and everything was working like before!

Sorry for the noise and I hope that it will help somebody running in the same problem!

2 Likes
3

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.