RUM Agent is missing on APM

Hello,

I have issue with RUM agent running on our front end servers. It is missing on APM (Kibana > APM > services). It was working at first, but when I checked APM after some time, all services with rum-js agents were missing. There was no upgrade on elastic stack, rum agent version remained at 5.3., there was also no change on apm-server.yml config (RUM part).

Kibana version : 7.8

Elasticsearch version : 7.8

APM Server version : 7.8

APM Agent language and version : Javascript & 5.3.0

Errors in browser console : (@elastic/apm-rum/dist/es/apm-base.js)

Uncaught (in promise) TypeError: r.isActive is not a function
    init apm-base.js:25
    default ampHelper.js:13
    onload index.js:38

Provide logs and/or server output :
in apm-server, there is repeating error (just this one)

2021-05-24T15:52:14.221+0200    ERROR   [request]       middleware/log_middleware.go:95 forbidden request       {"request_id": "aecb5571-bd2a-4aa6-a190-8dc7acc1dd89", "method": "POST", "URL": "/config/v1/agents", "content_length": 437, "remote_address": "10.10.162.113", "user-agent": "elasticapm-java/1.17.0", "response_code": 403, "error": "forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error."}

RUM agent config: (https://stuff.our.lab:8443/jsConfigVariables.js)

window.REACT_APP_AMP_CONFIG_ENABLED = true;
window.REACT_APP_AMP_CONFIG = {
  // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
  serviceName: 'stuff',
  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: 'https://stuff.our.lab:8443/apm/',
  // Set service version (required for sourcemap feature)
  serviceVersion: 'v87.2.0',
  //logLevel: 'debug'
};

APM-SERVER config export

apm-server:
  host: 0.0.0.0:8200
  rum:
    apm-server:
      rum:
        allow_origins:
        - '*'
        enabled: true
        source_mapping:
          enabled: true
          index_pattern: apm-*-sourcemap*
logging:
  files:
    keepfiles: 7
    name: apm-server
    path: /var/log/apm-server
    permissions: 384
    rotateeverybytes: 10485760
  metrics:
    enabled: false
  to_files: true
output:
  elasticsearch:
    enabled: true
    hosts:
    - elServer01.our.lab:9200
    indices:
    - index: apm-%{[observer.version]}-sourcemap
      when:
        contains:
          processor:
            event: sourcemap
    - index: apm-%{[observer.version]}-error-%{+yyyy.MM.dd}
      when:
        contains:
          processor:
            event: error
    - index: apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}
      when:
        contains:
          processor:
            event: transaction
    - index: apm-%{[observer.version]}-span-%{+yyyy.MM.dd}
      when:
        contains:
          processor:
            event: span
    - index: apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}
      when:
        contains:
          processor:
            event: metric
    - index: apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}
      when:
        contains:
          processor:
            event: onboarding
path:
  config: /etc/apm-server
  data: /var/lib/apm-server
  home: /usr/share/apm-server
  logs: /var/log/apm-server
setup:
  template:
    settings:
      _source:
        enabled: true
      index:
        codec: best_compression
        mapping:
          total_fields:
            limit: 2000
        number_of_shards: 1

Do you have any idea where should I look?

Thank you.

Hi @Hamunaptroid ,

Have you tried upgrading RUM agent to the recent versions? Could you try that out - RUM JS Agent version 5.x | APM Real User Monitoring JavaScript Agent Reference [5.x] | Elastic and let us know whether you are able to see them in APM UI.

Also, if the issue persists, Please change the logLevel: debug and post us the logs.

Thanks,
Vignesh

Hello @vigneshshanmugam, thank you for reply.
I would leave upgrading rum agent as a last option because we are keeping versions of agents and elastic components as mirror environment to production.

I have uncommented the loglevel: 'debug' in config file, but there is nothing new in element inspector on our web. Same errors when loglevel was commented out.
Do you mean different logs ?

[Script Loader] ReferenceError: require is not defined

<anonymous> 274.54d4fe24.chunk.js line 1 > eval:1

exports addScript.js:20

<anonymous> dSigXadesExtender.min.js:1

c status-report:1

<anonymous> index.js:1

c status-report:1

1625 helpers.js:6

c status-report:1

756 logo-aliter-blue.svg:1

c status-report:1

g status-report:1

e status-report:1

<anonymous> main.c6109df5.chunk.js:1

[addScript.js:8](https://stuff.our.lab:8443/portal/gui/static/node_modules/script-loader/addScript.js)
Uncaught (in promise) TypeError: r.isActive is not a function

init apm-base.js:25

default ampHelper.js:13

onload index.js:39

[apm-base.js:25](https://stuff.our.lab:8443/portal/gui/static/node_modules/@elastic/apm-rum/dist/es/apm-base.js)

Strict-Transport-Security: The connection to the site is untrustworthy, so the specified header was ignored.

It looks like rum-js agent configuration file "https://stuff.our.lab:8443/jsConfigVariables.js" is not working at all. Setting 'loglevel' or disabling it makes absolutely no difference to console output on web.

So if the config file is not working, that would explain the issues?

apm-server is not taking events from rum-js agent....