Help troubleshooting Elastic APM Integration: Installation Error - Input Not Supported

Elastic Observability APM
1

Kibana version: v 9.1.5
Elasticsearch version: 9.1.5
Elastic APM: 9.1.5

I'm having problems installing the Elastic APM integration on around half of my Fleet Agents. Getting the error: input not supported, have tried to research the message, but only found unanswered threads on this forum.

The Fleet Agents which ends up failing with input not supported all ends up as Unhealthy.

Not able to find any logs or other error messages explaining the problem I'm having. Also not able to discover what could be different between the Agents which is okey, and the failing ones.

Agent Logs:

event.dataset component.id message error.message
elastic_agent apm-default Spawned new component apm-default: input not supported -
elastic_agent apm-default Spawned new unit apm-default: input not supported -
elastic_agent apm-default Spawned new unit apm-default-22f49d32-9f95-4ea8-9ea5-5f8a062b4fc4: input not supported -

The goal is to install Elastic APM integration on all Elastic Agents and use the Java EDOT agent to gather OpenTelemetry data. As a workaround we have now installed one primary Elastic APM integration which all Java EDOT agent sends their data. However, we have experienced some data loss when our traffic is high, therefore thinking having a Elastic APM integration for each Elastic Agent is the proper solution.

Thoughts?

2

Hi @nilsen ,

could you share some details about the installation mechanism you chose? (docker, deb, etc). I suspect that you might have installed a basic flavor of the Elastic Agent for the instances where you receive the input not supported error. See Install Elastic Agents | Elastic Docs for more details. If you want to run apm integration, you would need to either run the server or complete flavor.

Created a github issue on our side to improve the returned error message https://github.com/elastic/elastic-agent/issues/11746.

1 Like
3

We used the auto generated script made by the fleet:

sudo ./elastic-agent install --url=https://myserver:8220 --enrollment-token=my-token==

It seems this is a basic flavor installation and might be the cause of the issue.

Tried now with --install-servers and it seems to work.

sudo ./elastic-agent install --install-servers --url=https://myserver:8220 --enrollment-token=my-token==

We still have no idea why it previously worked on around half of our agents, we tried to install the Elastic APM integration on around 10 fleet agents, and only half of them failed with input not supported. All of the Fleet agents have been install the same way, using the auto generated script made by the fleet. Could it be that sometimes the generated script includes --install-servers?

Is there a way we can check the Elastic Agent flavor? We still have some of the Agents running fine with the Elastic APM, could be interesting to see what flavor they are.

Thank you very much for your help.

4

you can check flavor by looking at {agent-dir}/data/elastic-agent-{hash}/components
if it includes only agentbeat, pf-host-agent and endpoint it’s basic deployment.

information is also visible in {agent-dir}/.flavor file.

i’d be curious what you find out.i havent seen fleet generating invalid command but aside from agents being older (as we introduced flavors only in 9.0) it is only reasonable explanation

5

Cannot find any {agent-dir}/.flavor, but checking the installed modules shows:

agentbeat
apm-server
cloudbeat
fleet-server
osqueryd
pf-host-agent

We are 101% sure we have only used the auto generated script made by the fleet, no idea why it would sometimes install a different flavor. The Elastic Agents and our fleet is fairly new, could it be that some are installed before 9.0 and they have gotten the required modules for APM?

Could I suggest adding a flavor option in Kibana when using the auto generated script made by the fleet?

6

what running ./elastic-agent version says.

as mentioned if you have mixed versions, agents <9.0 are expected to be fine and agent >=9.0 could fail with this flavor error.

you could check both version,flavor a component directory structure on agent that was ok and failing one

7

All our Elastic Agents are currently version 9.1.8 (all agents have been on the same version as each other), we keep them updated.

I was thinking maybe the agents that worked with the Elastic APM Integration was Elastic Agents installed pre 9.0, then later updated to >9. While the Elastic Agents that was installed post 9.0 was installed as basic flavor, therefore not working with Elastic APM Integration. Even if our installation approach was the same for all Elastic Agents.

We are currently working on upgrading/reinstalling all agents with the --install-servers tag, do you have anything else we should check before we override all our previously failing and working agents?

Thanks for all help.

8

this assumption is correct, if agent was installed pre-9.0 then we do not apply basic flavor to avoid losing capabilities.

nothing to add here, your procedure is correct

1 Like
9

Exactly, or stated another way.

In 8.x there is one flavor, full
In 9.x there are two flavors, full and slim, slim being the default

If you install a new 9.x Agent it is slim by default unless you pass --install-servers

If you upgrade an 8.x Agent to 9.x it is full by default but you can switch to slim for systems that do not need the server components to reduce disk footprint.

You are right that this creates some odd behavior in some cases, but the ultimate goal was to ensure that anything you upgrade continues to work. We’ve gone ahead and improved the error message when this happens to point to flavors as the likely cause: https://github.com/elastic/elastic-agent/pull/11825

1 Like