I know some of this might have been asked here before but Documentation and logs express differently.
I want to fetch logs from AWS Cloudwatch. For that, i'm using filebeat's input. This input is not doing any kind of multiline processing (this is not clear from the documentation either)
Since I can't do multiline "as close to the source as possible" I wanted to do it in Logstash. So I had a beats input with a multiline codec. But Logstash complains:
[ERROR] 2021-10-20 15:54:43.842 [[multilined-input]-pipeline-manager] beats - Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
[INFO ] 2021-10-20 15:54:43.848 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[ERROR] 2021-10-20 15:54:43.967 [[multilined-input]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"multilined-input", :exception=>#<LogStash::ConfigurationError: Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data.
Now, the documentation says that you should not use it:
If you are using a Logstash input plugin that supports multiple hosts, such as the beats input plugin, you should not use the multiline codec to handle multiline events. Doing so may result in the mixing of streams and corrupted event data. In this situation, you need to handle multiline events before sending the event data to Logstash.
I tried creating a single worker pipeline dedicated for this in order to prevent the mixing of streams but I can't get it to even start.
So, is it possible but not recommended, or not possible at all?
thanks!
Rodrigo.