Rsyslog and syslog-ng direct logging to Elasticsearch, viable replacement for elastic-agent?

Elastic Observability Logs
1

rsyslog has a module to send directly to Elasticsearch:

syslog-ng also has a module for logging directly to Elasticsearch:

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide

How viable are rsyslog and syslog-ng for sending directly to Elasticsearch?

Is these solutions viable replacements for elastic-agent?

I'm investigating alternatives to elastic-agent in very high load logging scenarios.

2

It depends on your use case and what you want to do with your logs.

No, they are not, Elastic Agent is not just a log collector, please read this answer on your other topic for more information.

You can use rsyslog/syslog-ng to send data to Elasticsearch, but you will need to create some ingest pipeline to parse your message, than you will need to create dashboards and alerts for your data.

Elastic Agent integrations already do that for you.

3

Thanks again for your very accurate response.

As I responded in the other thread: Using DataDog's vector to ship logs to ElasticSearch instead of elastic-agent? - #3 by Craig_Rodrigues

The main reason that I am looking for alternatives to Elastic Agent for shipping logs to elasticsearch was because of instability (zombie processes, high CPU load) that I saw in Elastic Agent 8.4.2.

4

Without troubleshooting this it is not possible to know if the issue was indeed with Elastic Agent, any log collector can have these issues depend on the amount of the data, specs of the machine etc.

5

Yes, the initial Elastic and Elastic-agent setup we had was very basic.
However between 8.4.2 and 8.9.0, Elastic Agent has gone through a lot of changes, as described in this presentation:
Evolution of the Elastic Agent

I didn't have time to fully investigate why elastic-agent 8.4.2 was running with zombie processes and high CPU load. At that time I had to just shut down elastic agent completely and not ship logs to Elastic.

The second Elastic setup we have is a large cluster set up in Kubernetes with ECK,
and we are paying special attention to memory usage and network load balancer on the Elastic cluster itself.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.