No idea. Have you confirmed your regex matching your logs (e.g. online regex tester, select golang)?
Any errors in filebeat logs?
Any other contents you didn't expect?
Skimming your configuration file, I'd say it looks good. You have a quite hughe (and duplicate) list of exclude_lines. Some way to reduce duplicates:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/{{ app_name }}/{{ app_name }}*.access.log
exclude_lines: ${app.shared.exclude_lines}
- type: log
enabled: true
paths:
- /var/log/{{ app_name }}/{{ app_name }}*.error.log
exclude_lines: ${app.shared.exclude_lines}
app.shared.exclude_lines: [
'go',
'beginning close',
'readLoop exiting',
'breaking out of writeLoop',
'writeLoop exiting',
'finished draining, cleanup exiting',
'clean close complete',
'connecting to nsqd',
'stopping',
'exiting router'
]
Given access/error log are mostly the same, why do you need 2 inputs? You plan to add some more meta-data per log type? Otherwise you can just put both paths into one input configuration:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/{{ app_name }}/{{ app_name }}*.access.log
- /var/log/{{ app_name }}/{{ app_name }}*.error.log
exclude_lines: [
'go',
'beginning close',
'readLoop exiting',
'breaking out of writeLoop',
'writeLoop exiting',
'finished draining, cleanup exiting',
'clean close complete',
'connecting to nsqd',
'stopping',
'exiting router'
]
Exclude lines is based on regexes (sometime being optimized to sub-string matches). If the contents to be exclude is at the beginning or end of the line consider to add ^ or $ to the regexes. This optimises the matches to into prefix/-suffix string matching (algorithm linear in time to pattern).
