Errors are not logged to APM

Elastic Observability APM
,
1

APM Agent language and version: javascript; '@elastic/apm-rum: 4.9.1'

Browser version: Any

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Cloud 7.1.1

Is there anything special in your setup?

  • Angular v 5.2.0

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

No error is logging to APM since 12AM UTC on Jun 11. No change was made on our side during this time. Throwing the error by force works for test Kibana (v7.6.2) but the same error is not logged to production (v 7.1.1)

Production Kibana:
Kibana version: v 7.1.1
Elasticsearch version: v 7.1.1
APM Server version: v 7.1.1

image
image1437×899 73.2 KB

image
image1850×528 38.2 KB

TEST Kibana
Kibana version: v 7.6.2
Elasticsearch version: v 7.6.2
APM Server version: v 7.6.2

image
image1434×895 43.8 KB

image
image1804×1010 127 KB

Steps to reproduce: Throw the generic error after app loads:

    setTimeout( () => {
      throw Error('Test Error');
    }, 1000 );
2

Hi @akosanovic,
could you please check the health of the error index setup. You can do so by querying GET apm-7.1.1-error/_ilm/explain. It should contain managed: true and no error information.

Please also check if any error events are ingested for your current version, by running GET apm-7.1.1-error/_count

3

HI @simitt,
Thanks for getting back,

I'm getting error 404 for apm-7.1.1 and apm-7.1.0

image
image683×730 89 KB

image
image891×672 94.6 KB

but it works ok on the test env

image
image990×774 45.3 KB

How do we fix this?

4

That is suprising. Have you changed any of the apm-server.yml configuration options and potentially disabled that the APM Server sets up templates and indices?
If not, please restart the APM Server, which should automatically take care of setting up the according ILM aliases, policies, etc and then try to run the _ilm/explain query again.

1 Like
5

We have tried restarting the APM Server but no changes.
We have a managed elastic cluster and apm server. We don't manage these ourselves.
We separate the apm indices in transactions, span and errors. And we create one index per day for each of these categories.
Transactions indices are still getting created daily but the span indices stopped getting created on the 12th and the errors ones on the 13th.

6

That sounds like you are not using ILM which would be the default since 7.5, but are manually setting the output.elastic.indices, in which case the above mentioned query will result in a 404. It also looks like you have been using ILM in 7.6 though. Could you confirm whether or not you are overwriting the default index settings and want to make use of ILM?
Can you check whether or not error events are generally ingested in 7.7 by running GET apm-7.7.1-error*/_count?

Since you are using an elastic managed cloud setup, you could also create a Help Issue on cloud for your deployment, mentioning APM ingestion issues, linking to this discuss post and mentioning me.

7

Hi @simitt,

I think that my last response Errors are not logged to APM caused misunderstanding, can you just ignore the screenshot for 7.6.

We are using only APM version 7.1.(1) upgraded from 7.0.0. Does that mean we don't have ILM by default? How is this affecting our logging since 12 AM UTC on Jun 11?

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.