LogStatsh- Grok Filter for Special character in the log file

[Bheeshma]

Hi,

I have the below log pattern in a single line and trying to filter this from massive log using GROK FIlter

2018-02-17 17:00:12.7280 Information |||| Application: Hyperion |||| User: Teddy |||| Template: Homepage |||| Report: FINREP |||| ExecutionTime: 200 ms

I have used this below grok filter however ( havent build the complete grok yet) however it couldn't identify how to filer the special Character |||| from this; could you please advise on how to capture the special characters |||| please

%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:Info}%{SPACE}%{QUOTEDSTRING}

from GROK Match Constructor:

2018-02-17 17:00:12.7280 Information |||| Application: Hyperion |||| User: Teddy |||| Template: Homepage |||| Report: FINREP |||| ExecutionTime: 200 ms

NOT MATCHED. The longest regex prefix matching the beginning of this line is as follows:

prefix %{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:Info}%{SPACE}
timestamp 2018-02-17·17:00:12.7280
Info Information
after match: |||| Application: Hyperion |||| User: Teddy |||| Template:
Homepage |||| Report: FINREP |||| ExecutionTime: 200 ms

Have used this pattern too but it is not working either;

%{TIMESTAMP_ISO8601:timestamp} %{WORD:Info} |||| %{WORD:Application} |||| %{USERNAME:user}

Thanks
Bheeshma.

[Bheeshma]

2018-02-17 17:00:12.7280 Information |||| Application: Hyperion |||| User: Teddy |||| Template: Homepage |||| Report: FINREP |||| ExecutionTime: 200 ms

With the help of http://grokdebug.herokuapp.com/ website, tried to build the grok filter upto 2018-02-17 17:00:12.7280 Information ||||

%{TIMESTAMP_ISO8601:timestamp} %{WORD:Info} %{SPACE}|%{NOTSPACE}|%{NOTSPACE}

and here is the match but unfortunately, It couldn't match the |||| so couldn't move proceed further; can you please review and help on how match the special characters |||| and complete the entire grok filter for the line

{
"timestamp": [
[
"2018-02-17 17:00:12.7280"
]
],
"YEAR": [
[
"2018"
]
],
"MONTHNUM": [
[
"02"
]
],
"MONTHDAY": [
[
"17"
]
],
"HOUR": [
[
"17",
null
]
],
"MINUTE": [
[
"00",
null
]
],
"SECOND": [
[
"12.7280"
]
],
"ISO8601_TIMEZONE": [
[
null
]
],
"Info": [
[
"Information"
]
],
"SPACE": [
[
""
]
],
"NOTSPACE": [
[
"|",
"|"
]
]
}

[Bheeshma]

Hi Gurus,

I am really newbie so really finding hard to understand how GROK works..I really need your advise how to achieve and match to the LOG details and produce accurate results which helps for diagnosing performance issues we have currently.

My apologies if I am nagging , just need small push and advise from this forum.

Kind Regards
Bheeshma.

[magnusbaeck (Magnus Bäck)]

Always format snippet from logs and configurations as preformatted text using the </> toolbar button. Otherwise we won't be able to see exactly what e.g. your grok expression looks like. Specifically, are you escaping the | characters with a backslash?

[Bheeshma]

Thanks Magnus for the reply; Here is the line which I have been trying to match and filter out from the massive log.
<
2018-02-17 17:00:12.7280 Information |||| Application: Hyperion |||| User: Teddy |||| Template: Homepage |||| Report: FINREP |||| ExecutionTime: 200 ms

/>

And this is the GROK filter trying to write to match to the above
<
%{TIMESTAMP_ISO8601:timestamp} %{WORD:Info} %{SPACE}|%{NOTSPACE}|%{NOTSPACE}

/>

Basically I am stuck how to escape the special character |.

Kindly advise.

Thanks
Bheeshma.

[magnusbaeck (Magnus Bäck)]

I repeat: Always format snippets from logs and configurations as preformatted text using the </> toolbar button.

Basically I am stuck how to escape the special character |.

Did you try with \|?

[Bheeshma]

This is how the log looks like :

2018-02-17 17:00:15.9105 Information |||| Application: P&L |||| User: Admin |||| Template: P&L |||| Report: P&L Report |||| ExecutionTime: 952 ms

And here is how wanted to view this data in Kibana and then generate the report depend on the name of the report and execution time it took

Timestamp 2018-02-17 17:00:15.9105
Info Information
Application: P&L
User: Admin
Template: P&L
Report: P&L Report
ExecutionTime: 952 ms

I tried several times to achieve the above result but ended up till here only which is not yielding any results.

<
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \s %{WORD:Info} %{WORD:Application} %{USERNAME} %{WORD:Template} %{WORD:Report}" }
}
}
/>

[magnusbaeck (Magnus Bäck)]

You're apparently ignoring what I am saying so perhaps you should try using the grok constructor site to get help coming up with a grok expression.

[Bheeshma]

Hi Magnus,

I really beg your pardon , not doing it intentionally; probably not getting or understanding how to report about the issue over here.

However, I tried this option and it worked and the entire message

" 2018-02-17 17:00:15.9105 Information |||| Application: P&L |||| User: Admin |||| Template: P&L |||| Report: P&L Report |||| ExecutionTime: 952 ms "

is appearing in Kibana web.

        filter {
            grok {
                  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}
                   }}

However, from this message, I want to filter out Report Name and the Execution time and populate in Kibana under Y & X axis.

is it possible to filter the Report name and the execution details from the message from Kibana -> Discover and then visualize in a pictorial representation?

Please advise.

thanks
Bheeshma.