Finding same pattern multiple times in one log file

Elastic Stack Logstash
1

Hi, I'm new to using the elk so the question may be easy / make no sense. One pattern appears 3 times with different numbers inside my log file. I can use the grok filter with the match config to find the pattern at all three instances. But I want the field to be called a different name every time this pattern is found so I can make 3 different visuals on kibana. For example I tried something like:

grok {
break_on_match => false
match => { "message" => "%{NUMBER:blah:int}"}
match => { "message" => "%{NUMBER:blah2:int}"}
match => { "message" => "%{NUMBER:blah3:int}"}
}

Thinking it would find the first instance -> call it blah, find second instance -> call it blah2, then find third instance -> call it blah3. But this doesn't work. Is there any way to do such a thing?

2

It's not quite clear from your question is these three occurrences appear in the same line or in different lines. Please show an example from your log file that exhibits the data you want to extract.

3

I was able to find a way to do this but thank you!

4

Can you share? It might be useful for others :slight_smile:

5

I still don't have a way to do it but I can use Kibana to add the sums of matches for that pattern which is good enough for me. I am using cucumber testing (u don't need to know what that is) on three components so the output, which is in one log file, is given the same way 3 different times. I can't really break the match up by components because of the way it is ran so I need a way to separate each output by giving it a different name.

Ex:
-> logs about running cucumber
-> ... (Useless logs)
-> "15 scenarios (1 failed, 2 skipped)"
-> logs about running cucumber
->... (Useless logs)
-> "6 scenarios (2 failed, 0 skipped)"
-> logs about running cucumber
->... (Useless logs)
-> "8 scenarios (3 failed, 1 skipped)"
-> ... (Useless logs)

I wanted to do something that says first time you find this pattern call it this: "%{NUMBER:firstTotal} scenarios ( %{NUMBER:firstFailed} failed, %{NUMBER:firstSkipped} skipped)".

Second time call it: this: "%{NUMBER:secondTotal} scenarios ( %{NUMBER:secondFailed} failed, %{NUMBER:secondSkipped} skipped.

Etc..
This way I can have separate fields for the 3 instances.

6

I have the same problem.
I am reading multiline from input and trying to match all the instances of the pattern. but it only output the first instance and not the rest. I'm not sure where I am making the mistake. this is my conf file content:
input {
file {
# change the path in your local to make it work
path => "/usr/local/src/logstash/log1.in"
start_position => beginning
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^show"
negate => true
what => next
}
}
}

filter {
grok {
# pay attention to the path
patterns_dir => ["./patterns/mypatterns"]
match => {"message" => ["(?m)configure vlan %{WORD:vlan} add ports %{PORTS:ports} %{TAG_INFO:tag_info}"]}
break_on_match => false
}
}

output {
stdout {
codec => rubydebug
}
}

and this is the content of log1.in
configure snmp sysName "NY_MPBN_SS_1"
configure snmp sysLocation "GSI New York"
configure snmp sysContact "JPoserio@globecommsystems.com"
configure timezone name New_Yor -240
configure sys-recovery-level switch reset

configure vlan CH_ACCESS add ports 19, 28, 46-47 tagged
configure vlan CN_GN add ports 10, 46-47 tagged
configure vlan CN_GN add ports 12 untagged
configure vlan CN_Gn_GSN_1 add ports 3-8, 46 tagged
configure vlan Default add ports 50 untagged
show switch

7

You should create your own thread for this question :slight_smile:

1 Like