Can we use codec multiline in o/p section with amazon_es plugin ?
Multiline codec is as far as I know for inputs only as any ordering is lost part that point. What is it you are trying to achieve?
- I cannot use multiline in input section because we have mixed log patterns and input is from s3 with all log pattern in single prefix s3 bucket .
2.I cannot use filter multiline since its deprecated
3.I choosed output since based on the field value I have to apply codec .
So in this case is it better to go with GROK?
A codec should be applied per file even if your plugin matches many files, so I do not understand why this is not possible. If the log entries that need to be grouped are not consecutive within a file it is not multiline processing you are after.
Correct.
At this point each line has been processed independently in different batches and multiple threads so any ordering has been lost. You can not apply multiline as an output codec.
How can I selectively apply codec in input section , s3 prefix is not possible since it doesnt support regex .
Can you provide an example of what your logs look like and what you are looking to achieve?
We have different type pf logs in s3 bucket, in that certain logs are multiline , So codec in input should be selective. Since s3 input doesnt support regex for prefix , do we have any alternative approach
auto_flush_interval doesn't seems to work for s3 input. Is this supported for s3 plugin?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.