Create multiple groks for different logs

Elastic Stack Logstash
1

Hey , I want to create a grok parser filter rules for 2 different logs . For example for nginx logs , a different grok parser rule and for apache a different grok parser in a same filter block of logstash.conf(/etc/logstash/conf.d/logstash.conf)
Can it be possible ? If yes please provide a solution.

2

Depends on Logstash version. Logstash 5.x can handle only one config file, 6.x can handle multiple ones. It makes a huge difference.
I haven't worked with 6.x yet, so I only know 5.x.
You could send the different logs on different ports, then you can tag the documents in the input plugin. You can then define conditionals based on the tags, but first, info on Logstash version would be good to know.

3

You could send the different logs on different ports, then you can tag the documents in the input plugin.

Whenever possible add tags or fields as close to the source as possible to avoid having to use different ports for different kinds of files. If you use Filebeat each prospector can add custom tags or fields that you can use in Logstash to select the right filter(s).

4

I defined nginx and squid logs in twi different prospector like below

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/nginx/access.log
      #- c:\programdata\elasticsearch\logs*

  • type: log
    enabled: true
    paths:

    • /var/log/squid/access.log
      fields:
      squid: true
      fields_under_root: true

But how I will parse these two different logs through a grok parser in logstash config file? What I will write in if condition?

5

Always format configuration snippets as preformatted text. Use Markdown notation. There's a toolbar button for it if you don't know Markdown.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.