Error sending data to APM server

[Prasanth_V]

Dear Team,

I'm facing the below issue, Could you please suggest me?

Note: We've a sufficient disk space and CPU core.

2021-09-21 12:07:05,978 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
  "accepted": 0,
  "errors": [
    {
      "message": "queue is full"
    }
  ]
}

[axw (Andrew Wilkins)]

Please see Common problems | APM Server Reference [7.14] | Elastic

[zx8086]

How would you do this for Fleet's APM Server as there is a limited amount of configuration options.

It would help if there was a away to disable irrelevant calls looking for a central configuration management, which is not currently possible with Fleet's APM module and Beats Instrumentation which is making these useful request to the APM server.

forbidden request: Agent remote configuration is disabled. Configure the apm-server.kibana section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the apm-server.rum section. If you are not using remote configuration, you can safely ignore this error.

[axw (Andrew Wilkins)]

How would you do this for Fleet's APM Server as there is a limited amount of configuration options.

@zx8086 great question! Currently there is only limited support for tuning output settings when running the Fleet/Agent integration. You can configure Elasticsearch output settings globally for all agents via Fleet settings:

We are currently investigating making APM Server more self-tuning, which will apply when running with or without Fleet.

It would help if there was a away to disable irrelevant calls looking for a central configuration management, which is not currently possible with Fleet's APM module and Beats Instrumentation which is making these useful request to the APM server.

Support for APM Agent central config was added to the integration in 7.14.0, so you should not be seeing those errors. Are you running 7.14.x?

[zx8086]

Support for APM Agent central config was added to the integration in 7.14.0, so you should not be seeing those errors. Are you running 7.14.x?

Running 7.14.2 across the board. It seems the errors are due to the Beats Agents and their APM Instrumentation, as this is the offending query captured below, communicating with the Fleet's Elastic Agents APM Service's response. In this case it for Auditbeat

# Instrumentation support for the packetbeat.
instrumentation:
    # Set to true to enable instrumentation of packetbeat.
    enabled: true

    # Environment in which packetbeat is running on (eg: staging, production, etc.)
    environment: "Development"

    # APM Server hosts to report instrumentation results to.
    hosts:
      - http://192.168.0.3:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    secret_token: "REDACTED"

There is not problem with the data getting into the APM Kibana dashboards, it just seems the Beat Agents are trying to get a configuration of sorts. 192.168.0.7 (GaalDornick)

[axw (Andrew Wilkins)]

Thanks for the details. I suspect we're just incorrectly logging an error because there's no APM agent central config defined. We'll take a closer look to verify and sort it out.

[axw (Andrew Wilkins)]

Actually, no need to look further into it. I just remembered I already fixed this recently: https://github.com/elastic/apm-server/pull/5917. This fix is in 7.15.0 which was just released. You should no longer see those errors logged after upgrading.

[zx8086]

Thanks, it just was jarring to seen 403 errors, but 7.15 seems to have cleared this up ! Well done for the fix