Cannot write Filebeat output to Elastic running on Docker Container on my MAC

I have created a 3 node Elastic Docker Container using the instructions from Elastic 8.2.3, and upgraded it to 8.7.1. I am able to successfully create objects in the Elastic database, and the docker-compose command successfully created the keystores so everything is running correctly as a standalone instance.

I installed Filebeats 8.7.1 on my MAC and using CEL as my filebeat.input and am able to succesfully write the output to the console using output.console. However, when I switch the output to output.elasticsearch, nothing is written to any index on my Docker Elasticsearch. I was able to accomplish this using the 7.17.9 version of Elastic in my Docker Container, but not in the v8. I receive the following error message in the filebeats logs:

{"log.level":"error","@timestamp":"2023-08-02T15:09:00.761-0500","log.logger":"esclientleg","log.origin":{"":"transport/logging.go","file.line":38},"message":"Error dialing x509: “es01” certificate is not standards compliant","":"filebeat","network":"tcp","address":"","ecs.version":"1.6.0"}

strong text

But I'm not sure if that is causing the error or not. If so, how can I bring my certificate into compliance?

Disregard, it turns out Filebeat v8x was populating Elastic via DataStream instead of Index. Everything is working.

I went through those steps too. If you don't configure it, it will automatically send to datastream.

Does your comment indicate you can configure it to write to an Index? The documentation didn't provide any options to DataStream.

Could you show you filebeat.yml ? (without credentials)

It turns out that I was expecting the Filebeat output to be in an index as was done in v7x. I have come to realize that in v8x Filebeat now writes to a Data Stream. I was able to find the Data Stream and it is writing as expected now. Thank you for your interest in this.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.