Is this config good ? Especially the regex one


#1

Hello dears,
I have a filebeat config as filebeat.yml
Could you take a look on it and check if there is a mistake or not.
Because, my filebeat is running correctly but when i check in Graylog2 web ui, i still find the 200 and 300 response code.
If you look on this lines

  • type: log
    enabled: true
    paths:
    - /var/log/nginx/.access.log
    exclude_lines: ['HTTP[^"]
    " (?:402|405|410|429|(2|3)[0-9]{2})']

200 and 300 response code should not be sent to Graylog2, right ?
Do i make some mistakes in my filebeat.yml ?
Thank you for your time dears.


(Steffen Siering) #2

No idea. Have you confirmed your regex matching your logs (e.g. online regex tester, select golang)?

Any errors in filebeat logs?

Any other contents you didn't expect?

Skimming your configuration file, I'd say it looks good. You have a quite hughe (and duplicate) list of exclude_lines. Some way to reduce duplicates:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/{{ app_name }}/{{ app_name }}*.access.log
  exclude_lines: ${app.shared.exclude_lines}
- type: log
  enabled: true
  paths:
    - /var/log/{{ app_name }}/{{ app_name }}*.error.log
  exclude_lines: ${app.shared.exclude_lines}

app.shared.exclude_lines: [
    'go',
    'beginning close',
    'readLoop exiting',
    'breaking out of writeLoop',
    'writeLoop exiting',
    'finished draining, cleanup exiting',
    'clean close complete',
    'connecting to nsqd',
    'stopping',
    'exiting router'
  ]

Given access/error log are mostly the same, why do you need 2 inputs? You plan to add some more meta-data per log type? Otherwise you can just put both paths into one input configuration:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/{{ app_name }}/{{ app_name }}*.access.log
    - /var/log/{{ app_name }}/{{ app_name }}*.error.log
  exclude_lines: [
    'go',
    'beginning close',
    'readLoop exiting',
    'breaking out of writeLoop',
    'writeLoop exiting',
    'finished draining, cleanup exiting',
    'clean close complete',
    'connecting to nsqd',
    'stopping',
    'exiting router'
  ]

Exclude lines is based on regexes (sometime being optimized to sub-string matches). If the contents to be exclude is at the beginning or end of the line consider to add ^ or $ to the regexes. This optimises the matches to into prefix/-suffix string matching (algorithm linear in time to pattern).


#3

I don't even think for this method exclude_lines: ${app.shared.exclude_lines} allowed :slight_smile:
Btw, thx for your suggestion... I fix the regex via regex101...
Thx so much for your time...


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.