APM Server returns 404

[leslie]

I set up es、kibana、apm-server and my django app months ago, and it works well. I can view apm analysis on kibana http://<kibana>/app/apm, and it help me to solve some performence issue.

But today, on kibana, I found the latest APM data is 20 days ago, and not more new data after it. I searched es apm-7.6.1-transactions-、apm-7.6.1-errors-, they all stop at the time.

Running journalctl -ex -u apm-server, it returns logs of error log, saying

May 15 20:58:56 iZbp1h0l1z8k2yu09tblfmZ apm-server[27138]: 2020-05-15T20:58:56.241+0800        ERROR        [request]        middleware/log_middleware.go:95        404 page not found        {"request_id": "2e54fea2-5a0c-4ec2-af8b-9b8b69221263", "method": "POST", "URL": "/v1/transactions", "content_length": 7762, "remote_address": "172.21.243.107", "user-agent": "elasticapm-python/3.0.5", "response_code": 404, "error": "404 page not found"}

so I tried curl http://apm-server/, it returns normal json

{
  "build_date": "2020-02-28T22:18:38Z",
  "build_sha": "3b7823bb329e0e5bfe25e106a3b93e8f61d0451f",
  "version": "7.6.1"
}

but both curl http://apm-server/v1/errors and curl http://apm-server/v1/transactions return 404.

I have try restart kibana、es、apm; re-install apm-server; even delete indices of apm-*，they made no difference. (other applications are using this es, i cannot delete all indices)

So why does it happen suddenly? can anyboby help me?

Kibana version: 7.6.1

Elasticsearch version: 7.6.1

APM Server version: 7.6.1

APM Agent language and version: python3 + django + elasticapm==3.0.5

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum 3.4.3

Fresh install or upgraded from other version? No

[beniwohli (Benjamin Wohlwend @ APM)]

Hi @leslie

The version of the agent that you use is not compatible with APM Server / Kibana 7.6. You can find more information on supported agent versions here: https://www.elastic.co/guide/en/apm/get-started/7.7/agent-server-compatibility.html

I recommend to upgrade to the newest version of the agent, 5.6.0. Note however that there were some backwards incompatible changes between major versions. You can read up on them in our upgrade guides, https://www.elastic.co/guide/en/apm/agent/python/current/upgrading-4.x.html and https://www.elastic.co/guide/en/apm/agent/python/current/upgrading-5.x.html

[leslie]

@beniwohli thanks a lot, it helped upgrading python agent to 5.6.0.

i have not idea why it works well before.

the git history shows it was using 3.0.5 all the time, maybe someone installed 5.6.0 and not commit it.

thanks again.

[beniwohli (Benjamin Wohlwend @ APM)]

That's great to hear!

Could it be that your APM Server / Kibana / Elasticsearch was updated from 6.x to 7.6 around the time it stopped working?

[leslie]

I am afraid not.

APM Server / Kibana / Elasticsearch was installed by me. I am sure they are 7.6 all the way.