APM Agent Settings Kibana Not Applying

[willis (Will)]

Kibana version:

7.9.2

Elasticsearch version:

7.9.2

APM Server version:

7.9.2

APM Agent language and version:

Ruby, 3.13.0

** Problem **

I am looking to use the Kibana APM Agent remote configuration to manage a Ruby APM agent. I have two applications with access to the apm-server that are correctly returning spans/data and presenting in Kibana. One agent (nodejs) is able to grab remote configuration settings from Kibana and apply them. The other agent (Ruby) does not seem to apply the configuration settings. You can see in the attached screenshot the first agent grabs the settings but the second (Ruby) always has "Not yet applied by any agents".

The apm-server logs show that the Ruby agent is asking for configuration settings:

2021-01-21T16:54:02.548Z INFO [request] middleware/log_middleware.go:97 request ok {"request_id": "9c3574c5-e41d-45b1-a5e7-acfd8c0c80eb", "method": "GET", "URL": "/config/v1/agents?service.name=*******", "content_length": 0, "remote_address": "100.96.234.12", "user-agent": "elastic-apm-ruby/3.13.0 http.rb/3.3.0 ruby/2.5.8", "response_code": 200}

The response code is always 200 from the apm-server.

[sqren (Søren Louv Jansen)]

Hi there,

The apm-server logs show that the Ruby agent is asking for configuration settings:

This does indeed sound odd. Since APM Server calls Kibana to get the configuration can you check the logs for Kibana also?
Specifically looks for requests to POST /api/apm/settings/agent-configuration/search. The log statements will most likely say something like:

Config was not found for my-service/my-environment

I'm interested to hear if requests from the ruby agent reaches Kibana or only requests from the node agent does.

[willis (Will)]

Thanks @sqren.

There are requests from the Ruby agent in Kibana to POST /api/apm/settings/agent-configuration/search that respond with

Config was found for c*****/undefined

However, my Ruby agent environment is not called undefined.

The nodejs has the correct environment name in the response:

Config was found for b*****/development

[sqren (Søren Louv Jansen)]

Thanks for the details.

This could be a bug. Would you mind creating a bug report on Github with the same details you already supplied here?