# \[Filebeat\] Metadata (Content-Type octet-stream) in S3 file - silently ignores 'Publish event'

**URL:** <https://discuss.elastic.co/t/filebeat-metadata-content-type-octet-stream-in-s3-file-silently-ignores-publish-event/286082>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 6, 2021, 11:53pm UTC](https://discuss.elastic.co/t/filebeat-metadata-content-type-octet-stream-in-s3-file-silently-ignores-publish-event/286082 "2021-10-06T23:53:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maksim](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@Maksim](https://discuss.elastic.co/u/Maksim)\
**Post date:** [October 6, 2021, 11:53pm UTC](https://discuss.elastic.co/t/filebeat-metadata-content-type-octet-stream-in-s3-file-silently-ignores-publish-event/286082/1 "2021-10-06T23:53:47Z")

</div>

First debug log:  
If Object-string like: `2021-10-06/2021-10-06T23:13:06.107Z-test@simulator.json`  
filebeat silently drop output

```auto
2021-10-06T23:13:08.827Z	DEBUG	[input.aws-s3.sqs]	awss3/sqs.go:77	Received 1 SQS messages.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxx"}
2021-10-06T23:13:08.828Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/sqs_s3_event.go:230	SQS message contained 1 S3 event notifications.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874"}
2021-10-06T23:13:08.828Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/s3_objects.go:123	Begin S3 object processing.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874", "bucket_arn": "xxxxxxxxx", "object_key": "2021-10-06/2021-10-06T23:13:06.107Z-test@simulator.json"}
2021-10-06T23:13:08.895Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/s3_objects.go:131	End S3 object processing.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874", "bucket_arn": "xxxxxxxxxx", "object_key": "2021-10-06/2021-10-06T23:13:06.107Z-test@simulator.json", "elapsed_time_ns": 67338130}
2021-10-06T23:13:08.895Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/sqs_s3_event.go:252	End processing SQS S3 event notifications.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxx", "message_id": "a13eca5d-26cf-4996-a214-595a26694874"}
2021-10-06T23:13:10.999Z	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2021-10-06T23:13:10.999Z	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2021-10-06T23:13:21.000Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 1m0s

```

Second log:  
Object-string like: `2021-10-06/test-file.json`

```auto
2021-10-06T23:20:18.914Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/sqs_s3_event.go:230	SQS message contained 1 S3 event notifications.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxx", "message_id": "5207184b-0685-42fa-9053-3f30fbf16d5b"}
2021-10-06T23:20:18.914Z	DEBUG	[input.aws-s3.sqs_s3_event]	awss3/s3_objects.go:123	Begin S3 object processing.	{"id": "B59ACEA85433728E", "queue_url": "https://sqs.eu-central-1.amazonaws.com/xxxxxxxxxxxxxxxx", "message_id": "5207184b-0685-42fa-9053-3f30fbf16d5b", "bucket_arn": "xxxxxxxxxxx", "object_key": "2021-10-06/test-file.json"}
2021-10-06T23:20:19.002Z	DEBUG	[processors]	processing/processors.go:203	Publish event: {
  "@timestamp": "2021-10-06T23:20:19.001Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.15.0",

```

---

<div class="post-metadata">

**Author:** ![Maksim](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@Maksim](https://discuss.elastic.co/u/Maksim)\
**Post date:** [October 7, 2021, 10:47pm UTC](https://discuss.elastic.co/t/filebeat-metadata-content-type-octet-stream-in-s3-file-silently-ignores-publish-event/286082/2 "2021-10-07T22:47:56Z")

</div>

_UPDATE:_  
**The problem is not with the filename**  
Issue was with **Metadata** (Content-Type):

- first log file (problem):  
`Content-Type	application/octet-stream`
- second (correctly processed):  
`Content-Type	application/json`

---

<div class="post-metadata">

**Author:** ![Maksim](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@Maksim](https://discuss.elastic.co/u/Maksim)\
**Post date:** [October 7, 2021, 11:15pm UTC](https://discuss.elastic.co/t/filebeat-metadata-content-type-octet-stream-in-s3-file-silently-ignores-publish-event/286082/3 "2021-10-07T23:15:51Z")

</div>

_filebeat.yml_ - pretty simple:

```auto
filebeat.inputs:
- type: aws-s3
  queue_url: https://sqs.xxxxxxxx
  access_key_id: xxxx
  secret_access_key: xxxx

output.console:
  pretty: true

```

Without `file_selectors` or any regexes processed settings

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 5, 2021, 1:16am UTC](https://discuss.elastic.co/t/filebeat-metadata-content-type-octet-stream-in-s3-file-silently-ignores-publish-event/286082/4 "2021-11-05T01:16:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.