Hi
maybe the subject is not clear. I will try to explain it.
I have a log file that looks like this
06/22/18 00:41:11.719 PID = 1036
...
06/22/18 00:41:11.757 host is "host1"
...
06/22/18 00:41:11.772 JOB ID: 87.113
...
06/22/18 00:41:12.813 user user1
...
06/22/18 01:52:52.075 status=0
...
06/22/18 01:53:23.099 PID = 17089
...
06/22/18 01:53:23.129 host is "host2"
...
06/22/18 01:53:23.145 JOB ID: 2539.0
...
06/22/18 01:53:23.234 user user2
...
06/22/18 02:51:08.274 status=0
...
The ... means other lines I am not interested in.
I want to print out every time I know the timestamp for a new event (matches line with PID), the host, the jobid and the user. And also I want to print out all of that again, plus the timestamp for end event (matches line with status) and the status itself. The rest of events can be dropped.
With a pattern file like this
START_TIMESTAMP %{DATE_US} %{TIME}
END_TIMESTAMP %{START_TIMESTAMP:endtime} (status=)%{NUMBER:status}
this code is working fine
filter {
grok {
add_tag => [ "matched", "starting" ]
match => { "message" => "%{START_TIMESTAMP:starttime} (?=PID =)" }
patterns_dir => ["/tmp/pat/"]
break_on_match => false
}
grok {
add_tag => [ "matched", "submithost" ]
match => { "message" => "(?<=host is \")%{HOSTNAME:submithost}" }
patterns_dir => ["/tmp/pat/"]
break_on_match => false
}
grok {
add_tag => [ "matched", "user" ]
match => { "message" => "(?<=user )%{WORD:user}" }
patterns_dir => ["/tmp/pat/"]
break_on_match => false
}
grok {
add_tag => [ "matched", "jobid" ]
match => { "message" => "(?<=JOB ID: )%{NUMBER:jobid}" }
patterns_dir => ["/tmp/pat/"]
break_on_match => false
}
grok {
add_tag => [ "matched", "finished" ]
match => { "message" => "%{END_TIMESTAMP}"}
patterns_dir => ["/tmp/pat/"]
break_on_match => false
}
if "matched" not in [tags] {
drop { }
}
if "starting" in [tags] {
ruby { code => '@@starttime = event.get("starttime")' }
} else {
ruby { code => 'event.set("starttime", @@starttime)' }
if "submithost" in [tags] {
ruby { code => '@@submithost = event.get("submithost")' }
} else {
ruby { code => 'event.set("submithost", @@submithost)' }
if "jobid" in [tags] {
ruby { code => '@@jobid = event.get("jobid")' }
} else {
ruby { code => 'event.set("jobid", @@jobid)' }
if "user" in [tags] {
ruby { code => '@@user = event.get("user")' }
} else {
ruby { code => 'event.set("user", @@user)' }
}
}
}
}
if "finished" not in [tags] {
if "user" not in [tags] {
drop { }
}
}
}
I get this output, which is what I expected.
{
"user" => "user1",
"submithost" => "host1",
"starttime" => "06/22/18 00:41:11.719",
"jobid" => "87.113"
}
{
"endtime" => "06/22/18 01:52:52.075",
"status" => "0",
"user" => "user1",
"submithost" => "host1",
"starttime" => "06/22/18 00:41:11.719",
"jobid" => "87.113"
}
{
"user" => "user2",
"submithost" => "host2",
"starttime" => "06/22/18 01:53:23.099",
"jobid" => "2539.0"
}
{
"endtime" => "06/22/18 02:51:08.274",
"status" => "0",
"user" => "user2",
"submithost" => "host2",
"starttime" => "06/22/18 01:53:23.099",
"jobid" => "2539.0"
}
However, if I try to re-write the logic in a simpler way, to avoid that log if-else structure, like this, it doesn't work anymore, but I don't see why.
[...]
if "starting" in [tags] {
ruby { code => '@@starttime = event.get("starttime")' }
}
if "submithost" in [tags] {
ruby { code => '@@submithost = event.get("submithost")' }
}
if "jobid" in [tags] {
ruby { code => '@@jobid = event.get("jobid")' }
}
if "user" in [tags] {
ruby { code => '@@user = event.get("user")' }
}
if "user" in [tags] {
ruby { code => '
event.set("starttime", @@starttime)
event.set("submithost", @@submithost)
event.set("jobid", @@jobid)
event.set("user", @@user)
'
}
}
if "finished" in [tags] {
ruby { code => '
event.set("starttime", @@starttime)
event.set("submithost", @@submithost)
event.set("jobid", @@jobid)
event.set("user", @@user)
'
}
}
[...]
Now all json docs in the output are almost equal to each other. It seems like I have created some global variables that override the fields for each line.
Maybe is not possible to do it this way, and only way is with the if-else-if-else... mechanism?
Any comment is more than welcome.
Thanks a lot in advance.