Parsing syslog messages with Different Network equipment providers

Elastic Stack Logstash
1

Hello All,

I am currently working on problem which involves setting up pipeline with Logstash for syslog events.
Here are the requirements
1: I need to parse syslog events , without identifying / marking them as syslog events before parsing.
I just need to take line as a string input which may or may not be syslog.

2: If its syslog, I need to identify whether it came from which vendor (cisco, arista, juniper etc)

3: I need to insert some vendor specific labels during filter process.

I have read Logstash documentation, I still do not get concrete idea of Grok filters. Documentation is not good and I dont understand even after using online grok debugger.

for eg :
If line is something like this :
Dec 8 23:12:40

what if I get extra spaces in between, will grok filters still able to parse the data ?
Dec 8 23:12:40

2

2: If its syslog, I need to identify whether it came from which vendor (cisco, arista, juniper etc)

Okay, sure. You could use something like this to have it try multiple grok expressions and add a tag that indicates which expression matched. You'll probably want to remove the _grokparsefailure tag at the end since otherwise most messages will have that tag.

filter {
  if "_grokparsefailure" not in [tags] {
    grok {
      match => ["message", "pattern for vendor 1"]
      add_tag => ["vendor1"]
    }
  }
  if "_grokparsefailure" not in [tags] {
    grok {
      match => ["message", "pattern for vendor 2"]
      add_tag => ["vendorN"]
    }
  }
  ...
  if "_grokparsefailure" not in [tags] {
    grok {
      match => ["message", "pattern for vendor N"]
      add_tag => ["vendorN"]
    }
  }
}

what if I get extra spaces in between, will grok filters still able to parse the data ?

The two example lines you posted look identical. Next time, format the paragraph as code to make consecutive spaces aren't collapsed.

Anyway, the answer is that it depends on the grok expression. It parses things according to your instructions.

3

Thanks a lot !!! I appreciate your help.

Line 1 : ABC "SPACE" PQR
Line 2: ABC "SPACE""SPACE""SPACE""SPACE""SPACE""SPACE" PQR

How can we parse this with predefined Grok patterns for Syslog ?
Also is this a right link to check all available grok patterns ? https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

I am using Logstash 2.1.0.

4

Use \s+ to match one or more spaces.

That file on GitHub contains the grok patterns in Logstash 1.4.2. They're more or less the same as in 2.1.0, but if you want to know the exact patterns supported by your Logstash you should look in your installation directory. For LS 2.1 the files are in /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns.

5

Thanks magnusbaeck !
Appreciate your quick help !