How do I use Elastic Agent to send log data to Logstash?

[Matt_Johnston]

Hello. Based on the documentation (Beats and Elastic Agent capabilities | Fleet and Elastic Agent Guide [8.6] | Elastic) I am under the impression that the Elastic Agent can send log data to Logstash. However, the only bit of documentation I found on configuring this setup is under the "Configure standalone Elastic Agents" section (Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic). Is it not possible to do this using Fleet-managed agents?

Also, it's unclear to me from the documentation how to actually get the Elastic Agent to send log data to Logstash. I'm assuming I need to update the elastic-agent.yml file. But how do I configure the file so that the Elastic Agent tracks a log file on my local machine as an input and then sends it to the Logstash output?

Any help on this matter would be greatly appreciated. Thanks!

[stephenb (Stephen Brown)]

Fleet Managed Agent with Logstash Output

Setup the logstash output

Then when you create a new Policy point that policy to the new Logstash Output

[Matt_Johnston]

Hi Stephen. Thanks for your response.

So, if I understand correctly, I will need to create a new agent policy that points to the Logstash Output. But since each agent can only be assigned a single policy does that mean that the agent will only send data to Logstash? I think that I had the impression that I could have a single elastic agent sending log data to Logstash (similar to how filebeats does it) but also sending tracing and metrics data straight to Elasticsearch. Is this not possible?

[stephenb (Stephen Brown)]

The Elastic Agent today has a single output per policy.

APM and traces send telemetry NOT through Elastic Agent / Nor fleet ... They are direct from APM Language clients.... to the APM Server Integration

But say you want logs to Logstash and Metrics direct to Elasticsearch from a Single Policy, that is not supported today as far as I understand, not sure when / if that is on the roadmap.

What you are describing is and output per integration... see if I can find out anything on that... but not today. For Fleet

For stand-alone integration I would need to look, I think it could be done but then that Agent is not Fleet Managed

[leandrojmp (Leandro Pereira)]

@Matt_Johnston

I think that I had the impression that I could have a single elastic agent sending log data to Logstash (similar to how filebeats does it) but also sending tracing and metrics data straight to Elasticsearch. Is this not possible?

The main difference is that filebeat is not managed as Elastic Agent is, also, Filebeat only supports one output and Elastic Agent share a lot of code with the beats.

This was not possible with Filebeat and it is also not possible with Elastic Agent.

[Matt_Johnston]

Okay, this is all good to know.

Although, I think you've blown my mental model out of the water a bit here. My understanding is that the APM integration is attached to an agent policy, which is assigned to the Elastic Agent. Thus, I thought that the Elastic Agent was responsible, perhaps indirectly, for sending tracing and metrics data through to Elasticsearch. But from your comments, it appears that this understanding is incorrect. Or am I missing something here?

[stephenb (Stephen Brown)]

APM Yup confusing... Study the Diagram...

Components and documentation | APM User Guide [8.11] | Elastic

APM agent ... little a is embedded with the app

APM Integration ... is the APM Server that typically runs as an Integration inside Elastic Agent with a Big A

@Matt_Johnston

Thus, I thought that the Elastic Agent was responsible, perhaps indirectly, for sending tracing and metrics data through to Elasticsearch.

No Elastic Agent Big A does not generate traces... it does collect them from the language agents or OTEL agents and forwards them to Elasticsearch

[Matt_Johnston]

Thank you. This is very helpful!