Filebeat to Elastic Agent(otel collector) migration - 9.4.2

Elastic Stack Elastic Agent
1

I am currently migrating our existing Filebeat-based log collection architecture to the Elastic Agent as OpenTelemetry Collector approach and would appreciate some clarification.
I am following the "Elastic Agent as OTel Collector" documentation and have been trying to replace our existing Filebeat pipelines with a configuration based on the filebeatreceiver.

current proof of concept is working partially:

  • filebeatreceiver successfully harvests Kubernetes container logs.
  • We can see events through the debug exporter.
  • Custom fields added by the Filebeat receiver processors are visible in the exported events.

However, we have not yet been able to successfully deliver these events using either:

  • Elasticsearch exporter
  • Logstash exporter

is this supported with 9.4.2?
filebeatreceiver -> (elasticsearch exporter | logstash exporter)

Can I have 2 active exporter for 1 service pipeline under Collectors:daemon:config:service:pipelines"

  • debug
  • logstash
  • elasticsearch
2

Hi @Ts_P

What are the errors from the exporter?

And it seems that you are leaving out components

What is the full Architecture

Elastic Agent (Non Otel / Legacy ) -> filebeatreceiver EDOT Collector / Upstream Collector -> elasticsearch exporter.

I would also suggest to share your otel.yaml for the gateway collector

3

@stephenb
No errors from Logstash or Elasticsearch exporter.
in debug exporter everything seems find .. events are successfully modified.. logs files are processed
it is deployed in k8s.
Collectors:daemon:config:service:pipelines

  • debug

  • logstash

  • elasticsearch

    I added a filebeatreceiver under:

    collectors:
  daemon:
    config:
      receivers:
        filebeatreceiver:
................
      exporters: 
        elasticsearch:
---------------
        logstash:
----------------

    using a Filebeat filestream input pointing to Kubernetes container logs:

    • type: filestream
    • paths: /var/log/containers/*.log
    • prospector.scanner.symlinks: true
    • multiline
    • exclude_lines

    I added my exporters here
    collectors:
    daemon:
    config:
    service:
    pipelines:
    logs:
    receivers:
    - filebeatreceiver
    exporters:
    - debug
    - elasticsearch

    I am following the elastic documentation and more precisely this (see the attached screenshot)

    image
    image1586×1347 181 KB

4

Apologies

it's still not clear to me what your full ingestion path is. You mentioned logstash I don't see a config.

I'm not sure I can help unless I know with the full ingestion architecture is.

And the full configs Little snippets are hard to reconstruct...

If you skip logstash and go straight from the collector to elasticsearch the do the events show up??

Do the events show up in logstash when you turn on the debug?

I'd like to help but you're going to need to be much more specific or perhaps someone else has an idea.

I do know putting logstash in the middle of an OTEL flow is prone to not working correct. Logstash is not really part of the native otel flow.

I don't know what indices you're trying to write to, if you've touched the data it could be rejected due to mapping conflicts, etc. too hard to tell without all the explicit detail.

Let start with this

Show Full Config for

Elastic Agent (Non Otel / Legacy ) -> filebeatreceiver EDOT Collector -> elasticsearch exporter -> Elasticsearch

When I get a chance I will try that as well

5

@stephenb my setup is quite straightforward...
as I said earlier if I use debug exporter(instead of elasticsearch or logstash) everything seems fine . Logstash is not requited I just wanted to test if I can send the data to logstash instead of elasticsearch . Again just replaced one exported with another ... nothing fancy,

The collector picked up the *.log files .
Processed them correctly.
Not sure why the exporters like elasticsearch and logstash does seems to work correctly.
In general is the setup shown in the screenshot (attached in my previous message) supported or not? That is the most important question.

Our main target is to migrate from Filebeat pipelines to Elastic agent as Otel collector.

I followed recommendations from this article:
Elastic Agent as an OpenTelemetry Collector | Elastic Docs

So the architectute is quite simple .

  1. Elastic agent as otel collector
    (I want to reuse filebeat configuration that is why we have filebeatreceiver section as in the screenshot). The documentations claims that it is supported. Also logs provided by debug exporter confirms it.
  2. The data is sent to different output based on the exporter set (debug , elasticsearch or logstash) .
The configuration file is massive but in general :

collectors:
  daemon:
    config:
	  receiver:
		  filebeatreceiver:
			  filebeat:
				inputs:
				  - type: filestream
					id: otel-filebeat-poc
					enabled: true
					paths:
					  - /var/log/containers/*.log
					prospector.scanner.symlinks: true
					exclude_lines: ['', '', '']
					multiline:
					  pattern: ''
					  negate: false
					  match: after
			  processors:
				- drop_fields:
					fields: ['..']
				- decode_json_fields:
					fields: ["message"]
					process_array: true
					target: "...."
					when:
					  not:
						regexp:
						  ......: "......."
					fields: ["message"]
					process_array: true
					target: ""
					add_error_key: true
					overwrite_keys: true
					max_depth: 4
				- drop_event:
					when:
					  and:
						- equals:
							log.level: '.....'
						- equals:
							service.name: '.....'
			  tags: ["...."]
			  fields:
				env: POC
				
			  fields_under_root: false
 

collectors:
  daemon:
    config:
	    exporters:
			debug:
			  verbosity: detailed
			logstash:
				hosts:
				  [
					"",
				  ]
				loadbalance: true
				ssl.certificate_authorities: ["/certs/tls-ca-bundle.pem"]
				ssl.verification_mode: "none"
			elasticsearch/otel:
			  timeout: 24h

don't pay attention to the indentation here. it is correct . just copy / paste caused some issues.
of course we have defines :
service: pipelines: logs: receivers

service: pipelines: logs: exported

6

Hi @Ts_P Sorry yeah I think I was confused about what you are asking
What seems simple / direct to one... is not always to another.....

So first and foremost I just ran this and it works as specified.
I am getting ECS logs in elasticsearch, basically the same as if used filebeat.

So yes the Right Hand Side of that image does work / supported.
Now if you are asking if every filebeat possible configuration works in the OTEL version I do not know that.

receivers:
    filebeatreceiver:
        filebeat:
            inputs:
                - data_stream:
                    dataset: generic
                  id: filestream-receiver
                  index: logs-generic-default
                  paths:
                    - /var/log/*.log
                  type: filestream
        queue.mem.flush.timeout: 0s

processors:
  batch: {}

exporters:
  elasticsearch:
    endpoints: ["http://localhost:9200"]
    mapping:
      mode: ecs

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver]
      processors: [batch]
      exporters: [elasticsearch]

This is NOT straight forward and does require "fancy" :slight_smile: ... there is no direct exporter to logstash AFAIK... you would have to do extra configuration etc...

It would be something along the lines of the following from an internal discussion ... not straightforward.

You can egress from collector w/ opentelemetry-collector/exporter/otlphttpexporter/README.md at main · open-telemetry/opentelemetry-collector · GitHub, encoding=json
. And then configure logstash for JSON/HTTP input.

Or you can egress using the elasticsearch exporter like this:

  elasticsearch/logstash:
    endpoints: # list of logstash endpoints (or load balancer)
      - http://logstash:8080
    flush:
      interval: 1s
    tls:
      insecure_skip_verify: true
    # organize output fields similar to filebeat
    mapping:
      mode: otel
    # required for es_bulk codec in logstash
    headers:
      Content-Type: application/x-ndjson

and then use the logstash http input w/ es_bulk encoding.

Hope this helps... in Short logstash is not currently part of a normal OTEL flow.
You are not the only enquiring / thinking about using logstash because of its processing capabilities.

7

thanks a lot for the hints .
I have managed to establish communication between Elastic agent and elasticsearch.
There was an issue with my env params. It was my fault. thanks for sharing your configuration.
Also I decided to pass my logs to exporter -> otpl_grpc/gateway and then the gateway send them to elasticsearch.
About logstash I wanna try this... it says it is compatible with filebeatreceiver

8

Good find not sure why I did not see it....

Pretty substantial caveat... It's not a generic logstash exporter

This component is only expected to work correctly with data from the Beat receivers: Filebeat receiver, Metricbeat receiver. Using it with data coming from other components is not recommended and may result in unex