I'm trying to set up a pipeline that takes log input from Filebeat, outputs the raw logs to Amazon S3, AND filters the logs to then store in elastic. Is there currently a way to configure the s3 output plugin to only output the raw message, whilst still outputting filtered message to elastic?
For a workaround, my next thought was to have two filebeat prospectors running at the same location, with each prospector assigning different tags. Then I could split upon the tag to decide which gets filtered and sent to elastic, and which gets outputted to s3. However, there seems to be issues with running multiple prospectors at the same file, so this is likely not feasible.
Any advice on my situation would be very much appreciated!