About AWS S3 Input

Elastic Stack Beats
,
1

Hi,
We're considering changes to the structure of our ingestion pipelines. One of those is to have dockerized services write their logs to s3 and then have filebeat read them back before forwarding.

Given that we have apps whose logs get split by docker would this work? Would filebeat be able to stitch logs back together which have been split by docker before being written to s3?

Thx
D

2

Do u have an example? What do u mean by split?

3

I'm also a bit confused by this "split" and "stitch together". Could you elaborate more on this, maybe provide some examples.

Here are some high level ideas:
If the logs have got some common fields, like the application name you can filter using those fields. Another option would be to add some extra fileds when reading the S3 files.

4

Hi,
I'm referring to the 16KB log line limit in docker. As we have apps which write a heavy payload we need to be able to recombine those messages.

Thx
D

5

Interesting. No filebeat has no way to combine multiple events into a single one. You'd have to use logstash probably to do that using memcache or something similar to store the intermediate data.

6

Well, filebeat can do it when docker writes to json files. It's unfortunate that this capability (recombine docker log messages) is only implemented in the docker input.

How the equivalent could be done in logstash isn't clear to me.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.