Negate/Ignore certain files

Elastic Stack Beats
1

Hi all,

I have a number of files in a folder called something similar to:

file1-auth-detail-20160101.gz
file1-auth-detail-20160102.gz
file1-auth-detail-20160103
file1-auth-detail-20160104

I want to ingest the files that aren't the gzips. In bash I'd just do something similar to ll /usr/local/var/log/radius/radacct/*auth*[!gz] to list out just the files I'm interested in, so I put the following in my filebeat.yml:

  • paths:
  - /usr/local/var/log/radius/radacct/*auth*[!gz]
document_type: radiusacct

However, this doesn't appear to be working. Is there anyway to achieve what I want without having to rename files or move others?

Cheers

2

Currently this is not possible. With 1.1 a new feature exclude_files and include_files will be introduce, which will make it possible to exculde all files ending with .gz for example. Have a look here: https://github.com/elastic/beats/pull/563

3

Awesome cheers for the reply. I think I've just hit upon a temporary solution in this case by using */usr/local/var/log/radius/radacct/auth-detail-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] but haven't tested it for more than a few minutes so far so will see if it holds!

4

As an alternative you can use the nightly build for 1.1 which should be quite stable: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/