I am trying to process a subset of elastic beanstalk logs with s3. The issue is that the path elastic beanstalk publishes to looks like this:
I have set up the config to use a prefix of "resources/environments/logs/publish/e-environment" because our environment is ephemeral so instance codes could change at any point but the environment code will remain the same.
The problem with this setup is that it tries to process every log in the path, but I only want to process a single type of log.
Ideally I'm trying to do something like this:
prefix => "resources/environments/logs/publish/e-environment/*/log.file.prefix"
But I can't see how to make that happen so right now my logstash config has a lot of overhead processing a bunch of files that don't contain lines I care about.
I'm sure there is a way to get what I want but I can't figure it out. Help?