Cannot read Cloudtrail logs using s3-sns-sqs plugin. Uncompleted message at the end of poller loop

When I try to ingest logs using the s3snssqs input, I see no events output but no errors either. If I enable DEBUG logs, I see an "Inside Preprocess" message followed by a "Payload in Preprocess" message and then the following message for each message read from the SQS queue?

[2020-10-07T18:45:56,328][DEBUG][logstash.inputs.s3snssqs ][main][xxxx] [Worker xxxx/0] uncompleted message at the end of poller loop. We´ll throw skip_delete. {:message_count=>2729}

These messages are being accessed using a cross-account role that has been verified to access both the s3 buckets as well as the sqs queue using the AWS CLI.

Here is the current LS config:

input{ 
        s3snssqs { 
            s3_role_session_name => "MY_ROLE_NAME" 
            region => "us-west-2" 
            s3_default_options => { "endpoint_discovery" => true } 
            role_arn => "arn:aws:iam::ACCOUNT_NUMBER:role/MY_ROLE_NAME" 
            queue => "QUEUE_NAME" 
            queue_owner_aws_account_id => "ACCOUNT_NUMBER" 
            type => "sqs-logs" 
            sqs_skip_delete => true 
            sqs_delete_on_failure => false 
            from_sns => true 
    } 
}

Any ideas what may be causing this?

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.