When I try to ingest logs using the s3snssqs input, I see no events output but no errors either. If I enable DEBUG logs, I see an "Inside Preprocess" message followed by a "Payload in Preprocess" message and then the following message for each message read from the SQS queue?
[2020-10-07T18:45:56,328][DEBUG][logstash.inputs.s3snssqs ][main][xxxx] [Worker xxxx/0] uncompleted message at the end of poller loop. We´ll throw skip_delete. {:message_count=>2729}
These messages are being accessed using a cross-account role that has been verified to access both the s3 buckets as well as the sqs queue using the AWS CLI.
Here is the current LS config:
input{
s3snssqs {
s3_role_session_name => "MY_ROLE_NAME"
region => "us-west-2"
s3_default_options => { "endpoint_discovery" => true }
role_arn => "arn:aws:iam::ACCOUNT_NUMBER:role/MY_ROLE_NAME"
queue => "QUEUE_NAME"
queue_owner_aws_account_id => "ACCOUNT_NUMBER"
type => "sqs-logs"
sqs_skip_delete => true
sqs_delete_on_failure => false
from_sns => true
}
}
Any ideas what may be causing this?