Filebeat not considering all file matched via wildcard (*)

[s8sg]

I'm using filebeat from an custom docker container.

Where /var/log is mounted as /data/system
and /var/lib/nomad/alloc as /data/nomad

I have the below files in the mounted directory:

1. /data/system
$ docker exec 4742ce2ddd47 ls -la /data/system

drwxr-xr-x    2 root     root             6 Feb 22 17:21 anaconda
drwxr-x---    2 root     root            94 Jul 26 02:31 audit
-rw-r--r--    1 root     root           935 Jul 25 06:00 boot.log
-rw-------    1 root     sshd       3091968 Jul 26 14:09 btmp
-rw-------    1 root     sshd         48384 Jul  1 02:52 btmp-20160701
drwxrwx---    2 167      167          16384 Jul 26 10:29 ceph
drwxr-xr-x    2 996      993              6 Nov 23  2015 chrony
-rw-r--r--    1 root     root         16003 Jul 25 06:00 cloud-init-output.log
-rw-r--r--    1 root     root         24078 Jul 25 06:00 cloud-init.log
-rw-r--r--    1 root     root         17676 Jul 26 14:01 cron
-rw-r--r--    1 root     root         35633 Jul  3 03:42 cron-20160703
-rw-r--r--    1 root     root         97698 Jul 11 03:48 cron-20160711
-rw-r--r--    1 root     root         47765 Jul 15 01:01 cron-20160717
-rw-r--r--    1 root     root         68905 Jul 25 03:38 cron-20160725
-rw-r--r--    1 root     root         36300 Jul 25 06:00 dmesg
-rw-r--r--    1 root     root         36141 Jul 19 07:28 dmesg.old
-rw-r--r--    1 root     root        292292 Jul 26 14:23 lastlog
-rw-------    1 root     root          1641 Jul 26 03:41 maillog
-rw-------    1 root     root          2364 Jul  2 03:46 maillog-20160703
-rw-------    1 root     root          6098 Jul 10 03:22 maillog-20160711
-rw-------    1 root     root          3064 Jul 14 03:28 maillog-20160717
-rw-------    1 root     root          3780 Jul 24 03:12 maillog-20160725
-rw-------    1 root     root     207185182 Jul 26 14:34 messages
-rw-------    1 root     root      11069008 Jul  3 03:41 messages-20160703
-rw-------    1 root     root      93373525 Jul 11 03:47 messages-20160711
-rw-------    1 root     root      93607378 Jul 15 01:17 messages-20160717
-rw-------    1 root     root     1251294825 Jul 25 03:38 messages-20160725
drwxr-xr-x    2 38       38               6 May 31 10:29 ntpstats
drwx------    2 root     root             6 Jun 10  2014 ppp
-rw-------    1 root     root        676596 Jul 26 14:26 secure
-rw-------    1 root     root        340811 Jul  3 03:41 secure-20160703
-rw-------    1 root     root       1834918 Jul 11 03:47 secure-20160711
-rw-------    1 root     root       1955397 Jul 15 01:17 secure-20160717
-rw-------    1 root     root       1242832 Jul 25 03:36 secure-20160725
-rw-------    1 root     root             0 Jul 25 03:38 spooler
-rw-------    1 root     root             0 Feb 22 17:12 spooler-20160703
-rw-------    1 root     root             0 Jul  3 03:42 spooler-20160711
-rw-------    1 root     root             0 Jul 11 03:48 spooler-20160717
-rw-------    1 root     root             0 Jul 17 03:21 spooler-20160725
-rw-------    1 root     root             0 Feb 22 17:10 tallylog
drwxr-xr-x    2 root     root            22 Jun 30 07:18 tuned
-rw-rw-r--    1 root     sshd        640512 Jul 26 14:26 wtmp
-rw-rw-r--    1 root     sshd       1325952 Jul  6 12:03 wtmp-20160707
-rw-------    1 root     root          6248 Jul 26 04:58 yum.log

2. /data/nomad
$ docker exec 4742ce2ddd47 ls -la /data/nomad

drwxr-xr-x    4 root     root            43 Jul 26 09:43 33af0f4c-55a9-41f0-70d4-5152ea4d4bbf
drwxr-xr-x    4 root     root            47 Jul 26 10:29 63356737-070f-c22c-3c03-0a7c1aae17d1

$ docker exec 4742ce2ddd47 ls /data/nomad/33af0f4c../alloc/logs

gslb-enduser-agent.stderr.0
gslb-enduser-agent.stdout.0

My Filebeat Configuration is defined as:

$ docker exec 4742ce2ddd47 cat filebeat/filebeat.yml

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    - paths: ["/data/system/*", "/data/nomad/*/alloc/logs/*"]
      input_type: log

output:
  elasticsearch:
    enabled: true

    hosts: ["172.31.9.202:9200", "172.31.4.4:9200", "172.31.2.180:9200"]

    index: "filebeat"

    template:

      name: "filebeat"
      path: "/filebeat/filebeat.template.json"
      overwrite: true

  # LogStash output is not used

# There are three options for the log ouput: syslog, file, stderr.
logging:
  to_syslog: false
  to_files: false

But in kibana I'm getting only below files in the table

/data/system/messages-20160703       1,944
/data/system/messages	             1,403
/data/system/secure-20160725	     1,153
/data/nomad/33af0f4c-55a9-41f0-70d4-5152ea4d4bbf/alloc/logs/gslb-enduser-agent.stderr.0	968
/data/system/secure-20160717  	    880
/data/system/messages-20160725      699
/data/system/messages-20160711	    238
/data/system/messages-20160717  	    156
/data/system/secure-20160711	    150
/data/system/secure	             72

Is there is anything wrong I'm doing ? Please Help.

Thanks in advance

[andrewkroh]

I think the issue is with the double wildcard used in the path. See https://github.com/elastic/beats/issues/2084

[s8sg]

Hi,

I have tested the same file path patterns manually by filepath.Glob() inside the container. It is considering all the files. Is there is any other setting that I'm doing wrong?

Thanks

[ruflin]

What do you mean by "tested manually"? Did you write a small Golang App?

[s8sg]

Yes.

But it was not an issue.
I was using kibana and it was only getting me 20 result as per the settings.

Thanks for the support

[ruflin]

@s8sg Nice. Any chance to share the code you wrote to compare it and do some additional testing. Perhaps you can directly chime in on https://github.com/elastic/beats/issues/2084 and add the details of your testing there.