[Filebeat] Metadata (Content-Type octet-stream) in S3 file - silently ignores 'Publish event'

First debug log:
If Object-string like: 2021-10-06/2021-10-06T23:13:06.107Z-test@simulator.json
filebeat silently drop output

2021-10-06T23:13:08.827Z	DEBUG	[input.aws-s3.sqs]	awss3/sqs.go:77	Received 1 SQS messages.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxx"}
2021-10-06T23:13:08.828Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/sqs_s3_event.go:230	SQS message contained 1 S3 event notifications.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874"}
2021-10-06T23:13:08.828Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/s3_objects.go:123	Begin S3 object processing.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874", "bucket_arn": "xxxxxxxxx", "object_key": "2021-10-06/2021-10-06T23:13:06.107Z-test@simulator.json"}
2021-10-06T23:13:08.895Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/s3_objects.go:131	End S3 object processing.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874", "bucket_arn": "xxxxxxxxxx", "object_key": "2021-10-06/2021-10-06T23:13:06.107Z-test@simulator.json", "elapsed_time_ns": 67338130}
2021-10-06T23:13:08.895Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/sqs_s3_event.go:252	End processing SQS S3 event notifications.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874"}
2021-10-06T23:13:10.999Z	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2021-10-06T23:13:10.999Z	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2021-10-06T23:13:21.000Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 1m0s

Second log:
Object-string like: 2021-10-06/test-file.json

2021-10-06T23:20:18.914Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/sqs_s3_event.go:230	SQS message contained 1 S3 event notifications.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxx", "message_id": "5207184b-0685-42fa-9053-3f30fbf16d5b"}
2021-10-06T23:20:18.914Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/s3_objects.go:123	Begin S3 object processing.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxxxxxx", "message_id": "5207184b-0685-42fa-9053-3f30fbf16d5b", "bucket_arn": "xxxxxxxxxxx", "object_key": "2021-10-06/test-file.json"}
2021-10-06T23:20:19.002Z	DEBUG	[processors]	processing/processors.go:203	Publish event: {
  "@timestamp": "2021-10-06T23:20:19.001Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.15.0",

UPDATE:
The problem is not with the filename
Issue was with Metadata (Content-Type):

  • first log file (problem):
    Content-Type application/octet-stream
  • second (correctly processed):
    Content-Type application/json

filebeat.yml - pretty simple:

filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.xxxxxxxx
  access_key_id: xxxx
  secret_access_key: xxxx

output.console:
  pretty: true

Without file_selectors or any regexes processed settings

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.