Help troubleshooting Elastic APM Integration: Installation Error - Input Not Supported

Elastic Observability APM
1

Kibana version: v 9.1.5
Elasticsearch version: 9.1.5
Elastic APM: 9.1.5

I'm having problems installing the Elastic APM integration on around half of my Fleet Agents. Getting the error: input not supported, have tried to research the message, but only found unanswered threads on this forum.

The Fleet Agents which ends up failing with input not supported all ends up as Unhealthy.

Not able to find any logs or other error messages explaining the problem I'm having. Also not able to discover what could be different between the Agents which is okey, and the failing ones.

Agent Logs:

event.dataset component.id message error.message
elastic_agent apm-default Spawned new component apm-default: input not supported -
elastic_agent apm-default Spawned new unit apm-default: input not supported -
elastic_agent apm-default Spawned new unit apm-default-22f49d32-9f95-4ea8-9ea5-5f8a062b4fc4: input not supported -

The goal is to install Elastic APM integration on all Elastic Agents and use the Java EDOT agent to gather OpenTelemetry data. As a workaround we have now installed one primary Elastic APM integration which all Java EDOT agent sends their data. However, we have experienced some data loss when our traffic is high, therefore thinking having a Elastic APM integration for each Elastic Agent is the proper solution.

Thoughts?

2

Hi @nilsen ,

could you share some details about the installation mechanism you chose? (docker, deb, etc). I suspect that you might have installed a basic flavor of the Elastic Agent for the instances where you receive the input not supported error. See Install Elastic Agents | Elastic Docs for more details. If you want to run apm integration, you would need to either run the server or complete flavor.

Created a github issue on our side to improve the returned error message https://github.com/elastic/elastic-agent/issues/11746.

3

We used the auto generated script made by the fleet:

sudo ./elastic-agent install --url=https://myserver:8220 --enrollment-token=my-token==

It seems this is a basic flavor installation and might be the cause of the issue.

Tried now with --install-servers and it seems to work.

sudo ./elastic-agent install --install-servers --url=https://myserver:8220 --enrollment-token=my-token==

We still have no idea why it previously worked on around half of our agents, we tried to install the Elastic APM integration on around 10 fleet agents, and only half of them failed with input not supported. All of the Fleet agents have been install the same way, using the auto generated script made by the fleet. Could it be that sometimes the generated script includes --install-servers?

Is there a way we can check the Elastic Agent flavor? We still have some of the Agents running fine with the Elastic APM, could be interesting to see what flavor they are.

Thank you very much for your help.

4

you can check flavor by looking at {agent-dir}/data/elastic-agent-{hash}/components
if it includes only agentbeat, pf-host-agent and endpoint it’s basic deployment.

information is also visible in {agent-dir}/.flavor file.

i’d be curious what you find out.i havent seen fleet generating invalid command but aside from agents being older (as we introduced flavors only in 9.0) it is only reasonable explanation