Filebeat to Elastic Agent(otel collector) migration - 9.4.2

Elastic Stack Elastic Agent
1

I am currently migrating our existing Filebeat-based log collection architecture to the Elastic Agent as OpenTelemetry Collector approach and would appreciate some clarification.
I am following the "Elastic Agent as OTel Collector" documentation and have been trying to replace our existing Filebeat pipelines with a configuration based on the filebeatreceiver.

current proof of concept is working partially:

  • filebeatreceiver successfully harvests Kubernetes container logs.
  • We can see events through the debug exporter.
  • Custom fields added by the Filebeat receiver processors are visible in the exported events.

However, we have not yet been able to successfully deliver these events using either:

  • Elasticsearch exporter
  • Logstash exporter

is this supported with 9.4.2?
filebeatreceiver -> (elasticsearch exporter | logstash exporter)

Can I have 2 active exporter for 1 service pipeline under Collectors:daemon:config:service:pipelines"

  • debug
  • logstash
  • elasticsearch
2

Hi @Ts_P

What are the errors from the exporter?

And it seems that you are leaving out components

What is the full Architecture

Elastic Agent (Non Otel / Legacy ) -> filebeatreceiver EDOT Collector / Upstream Collector -> elasticsearch exporter.

I would also suggest to share your otel.yaml for the gateway collector

3

@stephenb
No errors from Logstash or Elasticsearch exporter.
in debug exporter everything seems find .. events are successfully modified.. logs files are processed
it is deployed in k8s.
Collectors:daemon:config:service:pipelines

  • debug

  • logstash

  • elasticsearch

    I added a filebeatreceiver under:

    collectors:
  daemon:
    config:
      receivers:
        filebeatreceiver:
................
      exporters: 
        elasticsearch:
---------------
        logstash:
----------------

    using a Filebeat filestream input pointing to Kubernetes container logs:

    • type: filestream
    • paths: /var/log/containers/*.log
    • prospector.scanner.symlinks: true
    • multiline
    • exclude_lines

    I added my exporters here
    collectors:
    daemon:
    config:
    service:
    pipelines:
    logs:
    receivers:
    - filebeatreceiver
    exporters:
    - debug
    - elasticsearch

    I am following the elastic documentation and more precisely this (see the attached screenshot)

    image
    image1586×1347 181 KB

4

Apologies

it's still not clear to me what your full ingestion path is. You mentioned logstash I don't see a config.

I'm not sure I can help unless I know with the full ingestion architecture is.

And the full configs Little snippets are hard to reconstruct...

If you skip logstash and go straight from the collector to elasticsearch the do the events show up??

Do the events show up in logstash when you turn on the debug?

I'd like to help but you're going to need to be much more specific or perhaps someone else has an idea.

I do know putting logstash in the middle of an OTEL flow is prone to not working correct. Logstash is not really part of the native otel flow.

I don't know what indices you're trying to write to, if you've touched the data it could be rejected due to mapping conflicts, etc. too hard to tell without all the explicit detail.

Let start with this

Show Full Config for

Elastic Agent (Non Otel / Legacy ) -> filebeatreceiver EDOT Collector -> elasticsearch exporter -> Elasticsearch

When I get a chance I will try that as well