Filebeat not considering all file matched via wildcard (*)


(S8sg) #1

I'm using filebeat from an custom docker container.

Where /var/log is mounted as /data/system
and /var/lib/nomad/alloc as /data/nomad

I have the below files in the mounted directory:

1. /data/system
$ docker exec 4742ce2ddd47 ls -la /data/system

drwxr-xr-x    2 root     root             6 Feb 22 17:21 anaconda
drwxr-x---    2 root     root            94 Jul 26 02:31 audit
-rw-r--r--    1 root     root           935 Jul 25 06:00 boot.log
-rw-------    1 root     sshd       3091968 Jul 26 14:09 btmp
-rw-------    1 root     sshd         48384 Jul  1 02:52 btmp-20160701
drwxrwx---    2 167      167          16384 Jul 26 10:29 ceph
drwxr-xr-x    2 996      993              6 Nov 23  2015 chrony
-rw-r--r--    1 root     root         16003 Jul 25 06:00 cloud-init-output.log
-rw-r--r--    1 root     root         24078 Jul 25 06:00 cloud-init.log
-rw-r--r--    1 root     root         17676 Jul 26 14:01 cron
-rw-r--r--    1 root     root         35633 Jul  3 03:42 cron-20160703
-rw-r--r--    1 root     root         97698 Jul 11 03:48 cron-20160711
-rw-r--r--    1 root     root         47765 Jul 15 01:01 cron-20160717
-rw-r--r--    1 root     root         68905 Jul 25 03:38 cron-20160725
-rw-r--r--    1 root     root         36300 Jul 25 06:00 dmesg
-rw-r--r--    1 root     root         36141 Jul 19 07:28 dmesg.old
-rw-r--r--    1 root     root        292292 Jul 26 14:23 lastlog
-rw-------    1 root     root          1641 Jul 26 03:41 maillog
-rw-------    1 root     root          2364 Jul  2 03:46 maillog-20160703
-rw-------    1 root     root          6098 Jul 10 03:22 maillog-20160711
-rw-------    1 root     root          3064 Jul 14 03:28 maillog-20160717
-rw-------    1 root     root          3780 Jul 24 03:12 maillog-20160725
-rw-------    1 root     root     207185182 Jul 26 14:34 messages
-rw-------    1 root     root      11069008 Jul  3 03:41 messages-20160703
-rw-------    1 root     root      93373525 Jul 11 03:47 messages-20160711
-rw-------    1 root     root      93607378 Jul 15 01:17 messages-20160717
-rw-------    1 root     root     1251294825 Jul 25 03:38 messages-20160725
drwxr-xr-x    2 38       38               6 May 31 10:29 ntpstats
drwx------    2 root     root             6 Jun 10  2014 ppp
-rw-------    1 root     root        676596 Jul 26 14:26 secure
-rw-------    1 root     root        340811 Jul  3 03:41 secure-20160703
-rw-------    1 root     root       1834918 Jul 11 03:47 secure-20160711
-rw-------    1 root     root       1955397 Jul 15 01:17 secure-20160717
-rw-------    1 root     root       1242832 Jul 25 03:36 secure-20160725
-rw-------    1 root     root             0 Jul 25 03:38 spooler
-rw-------    1 root     root             0 Feb 22 17:12 spooler-20160703
-rw-------    1 root     root             0 Jul  3 03:42 spooler-20160711
-rw-------    1 root     root             0 Jul 11 03:48 spooler-20160717
-rw-------    1 root     root             0 Jul 17 03:21 spooler-20160725
-rw-------    1 root     root             0 Feb 22 17:10 tallylog
drwxr-xr-x    2 root     root            22 Jun 30 07:18 tuned
-rw-rw-r--    1 root     sshd        640512 Jul 26 14:26 wtmp
-rw-rw-r--    1 root     sshd       1325952 Jul  6 12:03 wtmp-20160707
-rw-------    1 root     root          6248 Jul 26 04:58 yum.log

2. /data/nomad
$ docker exec 4742ce2ddd47 ls -la /data/nomad

drwxr-xr-x    4 root     root            43 Jul 26 09:43 33af0f4c-55a9-41f0-70d4-5152ea4d4bbf
drwxr-xr-x    4 root     root            47 Jul 26 10:29 63356737-070f-c22c-3c03-0a7c1aae17d1

$ docker exec 4742ce2ddd47 ls /data/nomad/33af0f4c../alloc/logs

gslb-enduser-agent.stderr.0
gslb-enduser-agent.stdout.0

My Filebeat Configuration is defined as:

$ docker exec 4742ce2ddd47 cat filebeat/filebeat.yml


filebeat:
  # List of prospectors to fetch data.
  prospectors:
    - paths: ["/data/system/*", "/data/nomad/*/alloc/logs/*"]
      input_type: log

output:
  elasticsearch:
    enabled: true

    hosts: ["172.31.9.202:9200", "172.31.4.4:9200", "172.31.2.180:9200"]

    index: "filebeat"

    template:

      name: "filebeat"
      path: "/filebeat/filebeat.template.json"
      overwrite: true

  # LogStash output is not used

# There are three options for the log ouput: syslog, file, stderr.
logging:
  to_syslog: false
  to_files: false

But in kibana I'm getting only below files in the table

/data/system/messages-20160703       1,944
/data/system/messages	             1,403
/data/system/secure-20160725	     1,153
/data/nomad/33af0f4c-55a9-41f0-70d4-5152ea4d4bbf/alloc/logs/gslb-enduser-agent.stderr.0	968
/data/system/secure-20160717  	    880
/data/system/messages-20160725      699
/data/system/messages-20160711	    238
/data/system/messages-20160717  	    156
/data/system/secure-20160711	    150
/data/system/secure	             72

Is there is anything wrong I'm doing ? Please Help.

Thanks in advance :slight_smile:


(Andrew Kroh) #2

I think the issue is with the double wildcard used in the path. See https://github.com/elastic/beats/issues/2084


(S8sg) #3

Hi,

I have tested the same file path patterns manually by filepath.Glob() inside the container. It is considering all the files. Is there is any other setting that I'm doing wrong?

Thanks


(ruflin) #4

What do you mean by "tested manually"? Did you write a small Golang App?


(S8sg) #5

Yes.

But it was not an issue.
I was using kibana and it was only getting me 20 result as per the settings.

Thanks for the support :slight_smile:


(ruflin) #6

@s8sg Nice. Any chance to share the code you wrote to compare it and do some additional testing. Perhaps you can directly chime in on https://github.com/elastic/beats/issues/2084 and add the details of your testing there.


(system) #7

This topic was automatically closed after 21 days. New replies are no longer allowed.