Recursive glob patterns syntax help

Elastic Stack Beats
1

Hello,

I was attempting this pattern

- input_type: log
  paths:
    - /apps/log/**/catalina.out

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

  fields_under_root: true
  fields:
    service: my_service

To recursively search as the example states: /foo/** expands to /foo, /foo/*, /foo/*/*, and so on, but it did not work for me. I am guessing i have the syntax incorrect. I have used this in the past as apps/log/**/*.log ... that worked for me, but i only need the catalina.out logs in the example.

Thanks,

2

@cchooks2 Can you start filebeat with the following command:

./filebeat -c ./filebeat.dev.yml -v -e -d "prospector"

You should see how filebeat is expanding your path, I've tested on my side with the following structure:

ph@sashimi  /tmp/logs  tree
.
└── var
    ├── p
    │   └── c
    │       └── d
    │           ├── dont.log
    │           ├── e
    │           │   ├── dont.log
    │           │   └── findme.log
    │           └── findme.log
    ├── pa
    │   └── d
    │       └── e
    │           └── findme.log
    └── pxx
        └── d

With Filebeat configured with the following path /tmp/logs/var/**/findme.log, It will find all the files matching this glob pattern.

2018-01-31T14:52:21.429-0500    DEBUG   [prospector]    log/prospector.go:361   Check file for harvesting: /tmp/logs/var/p/c/d/findme.log
2018-01-31T14:52:21.429-0500    DEBUG   [prospector]    log/prospector.go:434   Start harvester for new file: /tmp/logs/var/p/c/d/findme.log
2018-01-31T14:52:21.429-0500    DEBUG   [prospector]    log/prospector.go:361   Check file for harvesting: /tmp/logs/var/pa/d/e/findme.log
2018-01-31T14:52:21.429-0500    DEBUG   [prospector]    log/prospector.go:434   Start harvester for new file: /tmp/logs/var/pa/d/e/findme.log
2018-01-31T14:52:21.429-0500    INFO    log/harvester.go:216    Harvester started for file: /tmp/logs/var/p/c/d/findme.log
2018-01-31T14:52:21.429-0500    DEBUG   [prospector]    log/prospector.go:361   Check file for harvesting: /tmp/logs/var/p/c/d/e/findme.log
2018-01-31T14:52:21.429-0500    DEBUG   [prospector]    log/prospector.go:434   Start harvester for new file: /tmp/logs/var/p/c/d/e/findme.log
3

@pierhugues,

I see the following when i run the command you provided:

2018/02/01 16:46:27.404657 crawler.go:38: INFO Loading Prospectors: 1
2018/02/01 16:46:27.404667 registrar.go:236: INFO Starting Registrar
2018/02/01 16:46:27.404730 sync.go:41: INFO Start sending events to output
2018/02/01 16:46:27.404786 prospector.go:83: DBG  File Configs: [/apps/log/**/catalina.out]
2018/02/01 16:46:27.404798 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018/02/01 16:46:27.404802 prospector_log.go:44: DBG  exclude_files: []
2018/02/01 16:46:27.404999 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2018/02/01 16:46:27.405145 prospector.go:124: INFO Starting prospector of type: log; id: 12907489179318276673
2018/02/01 16:46:27.405175 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2018/02/01 16:46:27.405225 prospector_log.go:70: DBG  Start next scan
2018/02/01 16:46:27.408302 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 0, After: 0
2018/02/01 16:46:37.408421 prospector.go:183: DBG  Run prospector
2018/02/01 16:46:37.408468 prospector_log.go:70: DBG  Start next scan
2018/02/01 16:46:37.413157 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 0, After: 0
2018/02/01 16:46:47.413346 prospector.go:183: DBG  Run prospector
2018/02/01 16:46:47.413372 prospector_log.go:70: DBG  Start next scan
2018/02/01 16:46:47.416433 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 0, After: 0
2018/02/01 16:46:57.399404 metrics.go:34: INFO No non-zero metrics in the last 30s
2018/02/01 16:46:57.416641 prospector.go:183: DBG  Run prospector
2018/02/01 16:46:57.416656 prospector_log.go:70: DBG  Start next scan
2018/02/01 16:46:57.419068 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 0, After: 0

As you can see it is not reading it ... files are actively being written.

Thanks

4

@pierhugues

When I set it back to /apps/log/*/*/catalina.out it picked up all the catalina.out logs. See a snippet below:

2018/02/01 16:54:44.021731 prospector_log.go:226: DBG  Check file for harvesting: /apps/log/aa/bc/catalina.out

2018/02/01 16:54:44.021738 prospector_log.go:259: DBG  Update existing file for harvesting: /apps/log/cc/dd/catalina.out, offset: 77138

2018/02/01 16:54:44.021999 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 172, After: 172

Thanks

1 Like
5

The glob is not working as apps/log/**/catalina.out, but is as apps/log/*/*/catalina.out.

Any thoughts as to why? The inputs are above.

Thanks

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.