Best Practices for Combining Fleet, Elastic Agent, Logstash, and Redis?

Elastic Stack Elasticsearch
1

Hi everyone,

I’ve been working with the Elastic Stack for several years and recently started exploring Fleet Server and Elastic Agent. While I appreciate the modern approach and prebuilt integrations, I’m struggling with several architectural limitations compared to my traditional setup.

My previous setup

I used this architecture:
Filebeat / Winlogbeat → Redis → Logstash → Elasticsearch

This gave me:

  • Decoupling and buffering via Redis (safe during outages),
  • Flexible parsing and routing in Logstash,
  • Multi-destination outputs from Logstash (e.g., Elasticsearch and external SIEMs),
  • Full control over pipelines and index templates.

Current challenges with Fleet + Elastic Agent

Now, everything seems designed to push logs directly into Elasticsearch ingest nodes. This brings some issues:

1. Loss of output flexibility

  • Elastic Agent doesn’t support multiple outputs
    In my previous setup, I could:
    • Fork specific logs to an external SIEM or store raw logs elsewhere.
    • Do that easily in Logstash, not possible with Elastic Agent alone.

2. Hard to use prebuilt integrations when logs don’t go directly to ES

  • Elastic’s integrations include dashboards, ingest pipelines, and index templates.
  • But they only work if the agent sends data directly to Elasticsearch.
  • If logs go through Logstash, I lose access to:
    • the pipeline definitions,
    • field mappings,
    • dashboards.

Yes, I know there are unofficial tools to convert ingest pipelines to Logstash configs — but conversions often end with errors. Manually exporting mappings and maintaining them is time-consuming and fragile.

3. Ingest Node vs. Logstash: limited capabilities

  • ES Ingest pipelines are fine for simple parsing, but:
    • Not as flexible as Logstash (e.g., conditional logic, branching).
    • Harder to extend and debug.
    • No easy way to test them incrementally.

So for custom application logs, I still prefer Logstash.

4. Lack of reliable buffering with Elastic Agent

This is the biggest issue for me.

In the old model:

  • Redis buffered all events.
  • I could safely take Logstash or ES offline temporarily.

Now

  • In the current Fleet + Elastic Agent setup, I’m unsure what happens if Elasticsearch is temporarily unavailable.
  • Specifically, I’d like to understand:
    • Is there any disk-based buffering, or is everything kept in memory?
    • What limits or settings define how much data the agent can buffer (e.g. size, time)?
  • In the past, I used Redis as a buffer, which worked well for short downtimes and traffic spikes.
  • I couldn’t find clear documentation or guidance on how Elastic Agent handles this kind of resilience today.

If events are lost during an outage, that’s a dealbreaker for production.

My current thoughts

  • For standard logs (e.g. Windows event logs), Fleet + Elasticsearch ingest nodes work well thanks to built-in integrations.

  • However, I lose the ability to forward logs to multiple outputs (e.g. SIEM, archive).

  • I don’t need to route metrics, but I do need to branch system logs in some environments.

  • Elastic Agent and Filebeat support only one output, so even basic use cases become limiting.

  • Running multiple agents per host just to overcome this is confusing and hard to maintain.

  • There’s no buffering layer like Redis between the agent and Elasticsearch — which previously helped handle:

  • short downtimes (e.g. during maintenance),

  • unexpected failures,

  • and log spikes.

My questions

What would be the recommended architecture if I want to:

  • Use Fleet / Elastic Agent for standard logs and integrations,

  • Use Logstash for parsing custom logs and routing to multiple destinations,

  • Keep reliable buffering (e.g., Redis or disk-based queue),

  • Prevent log spikes from directly impacting Elasticsearch cluster performance,

  • And still be able to forward selected logs (e.g., system logs collected via Elastic Agent) to additional destinations, not just Elasticsearch.

Any guidance or insight would be greatly appreciated.

Thanks!

2

This isn't an immediate help because it's not quite ready yet, but for the past several months we have been re-implementing the internals of Elastic Agent to use the OpenTelemetry collector for data collection. The biggest piece of this has been adding a way to run the Beat inputs+processors in the collector directly.

Pivoting Elastic's Data Ingestion to OpenTelemetry — Elastic Observability Labs has the details.

Once this is ready (can't give dates but we hope it's in the coming months), Elastic Agent for ingest cases covered by Beats is going to functionally be equal to using our Elastic distribution of the OpenTelemetry collector, where we or you (via customizing the distribution) can get additional exporters from upstream OpenTelemetry. We will also expose the collector's persistent queue.

If you wanted to use Elastic Agent today, you are limited to using Kafka for buffering or writing directly to Logstash. You can run ingest pipelines in Logstash, but you need an Enterprise license to do so. Elastic Integration filter plugin | Logstash Plugins

You could also try using Elastic Distribution of OpenTelemetry Collector | Elastic Distributions of OpenTelemetry (which is actually just Elastic Agent because they'll be the same thing once our architecture change is complete) or the upstream contrib collector distribution directly, possibly with the soon to GA Quickstart: Send data to the Elastic Cloud Managed OTLP Endpoint | Elastic Docs. The caveat to this is the data schema will be the OTLP schema and not ECS.

2 Likes
3

I think I can provide some help as I have a similar structure with Elastic Agent where I use Logstash and Kafka and migrated some data ingestion from Logstash pipelines to native integrations.

Quick question, do you have a paid license? Platinum or Enterprise? Depending on the license level you may have some limitations on how you can use Elastic Agent.

Now, everything seems designed to push logs directly into Elasticsearch ingest nodes .

Yes, Elastic Agent is mostly designed to ship data directly to Elasticsearch, which in many use cases would not be ideal, but it has 2 other outputs that may help, the Logstash output and the Kafka output.

Here the license level plays a big part, if you are on the basic license, you can have only one output for all your integrations, the exception is for the policy that has a fleet server, since Fleet Server can only ship directly to Elasticsearch, this output needs to be Elasticsearch, so you would need to have your Default output configured to Logstash, and this specific integration the output would be configured to Elasticsearch, I have this working on my on-prem cluster that is now with the Basic license since we migrated to the Cloud.

If you have a Platinum license, you can have different outputs per policy, so you could have a policy with agents that send data to Elasticsearch and other with agents sending data to Logstash or Kafka.

And if you have an Enterprise license, you can have a different output per integration, so in the same policy you can have integrations sending data to different places or with different configurations.

Hard to use prebuilt integrations when logs don’t go directly to ES

Not that hard, it is just not well documented, I have data flowing from Logstash into Elasticsearch and using the native ingest pipelines.

The main issue here is that once you decide to use an integration, you cannot change the raw message, it needs to arrive as it is in Elasticsearch.

I have multiple data sources working in this way, for example, Fortigate and Palo Alto devices ship data to a couple of logstash instances, that act as producers and send the data to Kafka topics, from there I have other logstash instances acting as consumers and getting those logs from Kafka and shipping it to Elasticsearch where they will be parsed by the correct ingest pipeline, my logstash pipeline looks like this:

# pipeline: fortigate consumer
# 
input {
    kafka {
        group_id => "siem-ls-fortigate"
        client_id => "${SIEM_LS_HOST}"
        bootstrap_servers => ["${SIEM_KAFKA}"]
        topics => ["fortigate"]
        decorate_events => false
        auto_offset_reset => earliest
        max_poll_records => 1500
        consumer_threads => 2
    }
}

output {
    elasticsearch {
        cloud_id => "${SIEM_CLOUD_ID}"
        cloud_auth => "${SIEM_CLOUD_AUTH}"
        compression_level => 3
        data_stream => true
        data_stream_type => "logs"
        data_stream_dataset => "fortinet_fortigate.log"
        data_stream_namespace => "fortigate"
    }
}

You could also use an Elastic Agent to do the same, in our case we keep Logstash because the Agent could not perform in the same way when reading from Kafka.

Ingest Node vs. Logstash: limited capabilities

Yeah, here we have the same issues, anything custom that we want more flexibility we cannot use Elastic Agent, we use Logstash or Vector, it is easier to implement, test and debug the parsing, also we may need to store the same data in different indices, this is not possible with ingest pipelines.

Lack of reliable buffering with Elastic Agent

Another issue that we also have, if you need to buffer events from Elastic Agent you need Kafka or Logstash, where you can use persistent queues or ship the data elsewhere.

3 Likes
4

Elastic Agent only supports a memory queue, if it cannot send data to the output, it will start filling up the memory queue, once it is full no new events are accepted.

It does not support disk queue, to not lost events you need to send the data to an output with more flexibility, like Logstash or Kafka, both are supported, Redis is not.

However, I lose the ability to forward logs to multiple outputs (e.g. SIEM, archive).
I don’t need to route metrics, but I do need to branch system logs in some environments.
Elastic Agent and Filebeat support only one output, so even basic use cases become limiting.
Running multiple agents per host just to overcome this is confusing and hard to maintain.

This can be solved using Logstash as the output where you can redirect the data to multiple destinations, send the same data to 2 or more different places.

You also cannot run multiple agents per host, at least not Fleet Managed agents, it is only one per host.

There’s no buffering layer like Redis between the agent and Elasticsearch — which previously helped handle:
short downtimes (e.g. during maintenance),
unexpected failures and log spikes.

This can be solved using Kafka as the output or Logstash also and redirecting the data to Kafka, which is what I prefer, I have logstash instances that act as producers and logstash/elastic agent instances that act as consumers.

What would be the recommended architecture if I want to:
Use Fleet / Elastic Agent for standard logs and integrations,
Use Logstash for parsing custom logs and routing to multiple destinations,
Keep reliable buffering (e.g., Redis or disk-based queue),
Prevent log spikes from directly impacting Elasticsearch cluster performance,
And still be able to forward selected logs (e.g., system logs collected via Elastic Agent) to additional destinations, not just Elasticsearch.

If there is an integration and you want to use it, send the data directly to Elasticsearch, if you do not want to use the integration or need to send the data to a differente place, you need to use Logstash, if you want to have buffering, you can send directly to Kafka or to any other tool through Logstash.

I think that if you ditch Redis and replace it with Kafka, you will be able to solve the majority of your issues.

3 Likes
5

In a prior environment, I also used filebeat and winlogbeat sent via logstash. I had the beat modules that use ingest pipelines using logstash. I don’t have the configs, but logstash output section checks to see if the pipeline name exists and if it does, passes that to logstash.

The negative is you must control the beats version in use and pre-load the pipelines before updating beats version.

1 Like
6

Hello Craig @cmacknz,
thanks a lot for the detailed explanation — it’s really encouraging to see the direction Elastic is taking.

The move toward full OpenTelemetry compatibility, support for upstream exporters, and exposing the persistent queue is exactly what I was hoping for.

It’s great to see the Elastic Agent becoming more flexible and open. Appreciate the transparency about where things stand today and what’s coming — I’m looking forward to trying it out soon!

7

Thank you so much Leandro @leandrojmp for your detailed and thorough response — it really helped me understand the current limitations and best practices. I appreciate you taking the time to explain everything so clearly!