Logstash s3-sqs input plugin to differentiate 2 types of log

Elastic Stack Logstash
1

Hi all,

I have an use case where application logs are pushed to s3 bucket, log types are json and plain. Am using [logstash-input-s3-sns-sqs] plugin to read logs from S3 bucket. Both log types are pushed to different path in same bucket
Json log path : <my_bucket>/jsonlog/
plain log path: <my_bucket>/plainlog/

I am using below input filter and trying to separate these logs but it does not seem to work. Any thoughts on this is welcomed.

    input {
      s3snssqs {
        region                     => "us-east-1"
        s3_default_options         => { "endpoint_discovery" => true }
        queue                      => "<sqs_url>"
        from_sns                   => false
        s3_options_by_bucket       => [
            { bucket_name => "<my_bucket>"
            folders => [
            { key => ".*app1.log"
            codec => "json"
            type => "myjsonlog"},
            { key => ".*app2.log"
    	codec => "plain"
            type => "myplainlog"}]
            }
            ]
    }
    }

Thanks,
Vinod kumar

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.