# S3 input issues with images

**URL:** <https://discuss.elastic.co/t/s3-input-issues-with-images/122999>\
**Category:** Logstash\
**Created:** [March 8, 2018, 4:22am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999 "2018-03-08T04:22:49Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 8, 2018, 4:22am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/1 "2018-03-08T04:22:49Z")

</div>

I've been trying to figure out how to solve this: I have an Amazon S3 bucket with a lot of files and filetypes. I only care about the xml files. I can't figure out how to have it only process xml files. It keeps trying to process every file and because of that the images keep causing charset errors.

Is there a way to deal with this?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2018, 7:39am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/2 "2018-03-08T07:39:36Z")

</div>

The `exclude_pattern` or `prefix` options can't help?

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 8, 2018, 12:25pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/3 "2018-03-08T12:25:24Z")

</div>

`prefix` can't since the only pattern is the suffix of `.xml`. I tried using `exclude_pattern` but looking at the logs I didn't see it output anything using `stdout`. Am I right in thinking `exclude_pattern` is for the filename? It says "key" in the docs and I wasn't sure if that was the same thing.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2018, 1:41pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/4 "2018-03-08T13:41:59Z")

</div>

`exclude_pattern` is matched against the filename.

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 8, 2018, 1:57pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/5 "2018-03-08T13:57:19Z")

</div>

Here is what I'm using. Am I correct in thinking this should go through every file in S3 and process any file whose filename ends in `.xml`?

```
input {
  s3 {
    // credentials omitted
    exclude_pattern => "^((?!xml$).)*$"
  }
}

#filter {                                                                                                                                   
# xml {                                                                                                                               
# source => "message"                                                                                                      
# store_xml => false                                                                                                               
# }                                                                                                                                   
#}                                                                                                                                          

output {
  stdout {
    codec => rubydebug {
      metadata => true
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2018, 2:34pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/6 "2018-03-08T14:34:12Z")

</div>

No, that expression doesn't look right. Wouldn't `(?!\.xml)$` stand a better chance? But negative regexp assertions isn't my forte.

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 8, 2018, 2:50pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/7 "2018-03-08T14:50:50Z")

</div>

I'm thinking the issue might be the regex is wrong as well. I tried yours and it isn't processing anything either.

Isn't matching file types / extensions something that most people would need to do with Logstash? I feel like it would be very useful to have a built in way to say "only process files with this or that type" on the level of a codec or something.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2018, 3:13pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/8 "2018-03-08T15:13:33Z")

</div>

> Isn't matching file types / extensions something that most people would need to do with Logstash? I feel like it would be very useful to have a built in way to say "only process files with this or that type" on the level of a codec or something.

It's not an unreasonable request; feel free to file a GitHub issue.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2018, 3:35pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/9 "2018-03-08T15:35:58Z")

</div>

> [@arisbanach](#):
>
> I'm thinking the issue might be the regex is wrong as well.

\.(?!xml) (without the anchor) matches the way you would expect, even when the .xml is non-terminal. Which, for me, is unexpected. 🙂

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 8, 2018, 4:51pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/10 "2018-03-08T16:51:25Z")

</div>

@Badger Thanks! Is this site not working correctly though? [http://rubular.com/r/yP89qfpIBB](http://rubular.com/r/yP89qfpIBB)

I tested it out there and it doesn't seem to be working.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 8, 2018, 5:19pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/11 "2018-03-08T17:19:21Z")

</div>

> [@arisbanach](#):
>
> I tested it out there and it doesn't seem to be working.

Me too. [https://regexr.com/](https://regexr.com/) worked for me.

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 8, 2018, 5:49pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/12 "2018-03-08T17:49:56Z")

</div>

Great, I will try this out. Thanks!

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 9, 2018, 1:03am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/13 "2018-03-09T01:03:38Z")

</div>

Looks like the site you used uses JavaScript regex, but the S3 input plugin requires Ruby regex. So complicated, and still not working. It's odd that I can't find anyone else who has had this issue.

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 9, 2018, 1:39am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/14 "2018-03-09T01:39:57Z")

</div>

This works as expected for me when I check on Rubular: `^((?!XML$).)*$`

However, @magnusbaeck I am running Logstash with this:

```
exclude_pattern => "^((?!XML$).)*$"

```

and it shows nothing in the logs when I have this set as the output:

```
output {
  stdout {
    codec => rubydebug {
      metadata => true
    }
  }
} 

```

I've confirmed that there are .XML files in the bucket. It doesn't matter that they're nested several directories in, does it?

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [March 12, 2018, 11:09pm UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/15 "2018-03-12T23:09:08Z")

</div>

Also, I am somewhat confused about how the regex `exclude_pattern` field is supposed to work. For example, if I have regex that matches _part_ of the filename but not the entire thing, will that file be excluded? Does it have to match the entire filename to take effect?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 13, 2018, 6:49am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/16 "2018-03-13T06:49:36Z")

</div>

I don't think there are any implicit anchors, i.e. if the given expression matches the filename string that file is excluded. So partial match, if you will.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 10, 2018, 6:50am UTC](https://discuss.elastic.co/t/s3-input-issues-with-images/122999/17 "2018-04-10T06:50:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.