Grok filtering in logstash for multiple defined patterns

Elastic Stack Logstash
1

I am trying to filter my logs matching few patterns I have in the field "full_log" e.g:

E/vincinity/dholland_view_sql_global/IN/Cluster_Node/SSL-CACHE/Dsal1
F/vincinity/dholland_view_sql_local/IN/Cluster_Node3/SSL-CACHE/Dsal4
R/vincinity/dholland_view_sql_bran/IN/Cluster_Node/Sample/vr1.log

Now I want to grep these 3 paths from a bunch of logs: basically the pattern that I want to extract is logs containing "vincinity" "sql" and "IN" so with regex it would be simply vincinitysqlIN I tried this grok filter:

grok {

    match => { "full_log" => "%{URIPATHPARAM:*vincinity*sql*IN*}" }

  }

Then I get _grokparsefailure in kibana - I'm brand new to grok, so perhaps I'm not approaching this correctly.

Thanks,

2

You should read an introductory article on regular expressions (which are what grok is based on). To match multiple occurrences of any character, use .* not *.

Secondly, grok patterns are used like this: %{PATTERN_NAME:field}, where field is the name of the field in which to store the matched text. I don't think you want stuff stored in a field named *vincinity*sql*IN*.

I don't know what you're trying to do. Do you want to extract particular substrings from the contents of the full_log field? Or ignore all events whose full_log field don't contain vincinity, sql, and IN?

3

Thanks this is really helpful - but I'm now having a grok exception, and I'm actually looking for full_log messages containing the path .*vincinity.*sql.IN.

here's the config:

cat ./patterns/vinc.txt
VIN .*vincinity.*sql.*IN.*

logstash grok conf:

grok {
     patterns_dir => "./patterns"
     match => ["full_log" , "%{VIN:path}"]
}
4

I'm not sure you should be using the grok filter at all. Unless you explain what you want to accomplish it's very hard to give relevant help.

I'm actually looking for full_log messages containing the path ...

Yes, and then what? What do you want to do with those events? What do you want to do with the other events?

5

Okay so I need to tag full_log fields containing .*vincinity.*sql.IN. so that I can perform a search in Kibana using that tag.
This is one example but if it works I'll apply more customized patterns later and have more tags for the full_log fields.

6

Then I suggest something like this:

if [full_log] =~ /vincinity.*sql.*IN/ {
  mutate {
    add_tag => ["whatever"]
  }
}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.