Logstash monitoring BB-Server Logs


(Lukas Meier) #1

hi my mission it is to monitor a Blackberry (BES) Server with logstash
i got 3 types of Logfiles on BES-Server to parse,
core, mdat and disp logfile-types,
but they save wrong into ES,
with 2 types it was more easier,
but now with 3,
i dont know how to solve the prob,
someone can help pls?

logstash:
input {
lumberjack {
# The port to listen on
port => 5000

# The paths to your ssl cert and key
ssl_certificate => "/opt/logstash/etc/log.x.crt"
ssl_key => "/opt/logstash/etc/logstash.key"
    }

}

filter {
if [type] == "bb_core" {
grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601}%{SPACE}%{NOTSPACE}%{SPACE}%{PROG:cLevel} {%{SYSLOG5424PRINTASCII:cThread} : %{WORD:cQUEUE}} %{SYSLOG5424PRINTASCII:cNone} %{SYSLOG5424PRINTASCII:cCorrelation} - %{PROG:cStatus} %{GREEDYDATA:cMessage}" }

  			}

if "_grokparsefailure" in [tags] {
mutate {
type => "bb_disp" }
}

	}

if [type] == "bb_disp" {
grok {
match => { "message" => "\A<#%{INT:dRautenID}>[%{INT}] (%{NOTSPACE:dDate} %{HAPROXYTIME:dTIME}%{NOTSPACE:dID2} {%{JAVAFILE:dUsername}} %{CISCO_REASON:dAction}(%{WORD}%{NOTSPACE} %{NOTSPACE:dID}%{SPACE}%{WORD}=%{NOTSPACE:dEmail} %{HOST}=%{WORD:dDeviceID};%{UUID:dCorrelation}, %{WORD}=%{WORD:dRouting}, %{USER}=<%{WORD:dAgent}>, %{USER}=%{WORD:dTime}, %{WORD}=%{INT:dExt}, %{WORD}=%{INT:dwl}%{NOTSPACE}%{SPACE}%{GREEDYDATA:dMessage}"}
}

if "_grokparsefailure" in [tags] {
mutate {
type => "bb_mdat" }
}
}

if [type] == "bb_mdat" {
grok {
match => { "message" => "\A<%{TIMESTAMP_ISO8601}%{SPACE}%{WORD}>:[%{INT:mID}]:<%{USERNAME:mDOMAIN}>:<%{WORD:mAction}>:<%{WORD:mLayer} = %{WORD:mLayerinfo}, %{GREEDYDATA:mMessage}"}
}
}

if "_grokparsefailure" in [tags] {
mutate {
type => "bb_core" }
}

}

output {
if "_grokparsefailure" not in [tags] {
elasticsearch { host => localhost }
stdout { codec => rubydebug } } }

forwarder:

{
"network": {
"servers": [ "log.x:5000" ],
"ssl ca": "/opt/logstash-forwarder/etc/log.x.crt",
"timeout": 15
},
"files": [
{
"paths": [
"/var/log/bbx1.log"
],
"fields": { "type": "bb_core" }
},

{
"paths": [
"/var/log/bbx1.log"
],
"fields": { "type": "disp" }
},

{
"paths": [
"/var/log/bbx1.log"
],
"fields": { "type": "bb_mdat" }
}
]
}


(Lukas Meier) #2

i think the patterns are right,
why its saving into wrong logtype?

"message" => "2015-09-29T11:47:08.876+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG ",
"@version" => "1",
"@timestamp" => "2015-10-01T08:55:15.350Z",
"file" => "/var/log/bbx1.log",
"host" => "s99",
"offset" => "409",
"type" => "bb_core",
"cLevel" => "CORE",
"cThread" => "wff-thread-98",
"cQUEUE" => "loadDirGroupsUsersQueueConsumer",
"cNone" => "none|utility",
"cCorrelation" => "[{}]",
"cStatus" => "DEBUG"
}
{
"message" => "2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG ** making TGT request .. **",
"@version" => "1",
"@timestamp" => "2015-10-01T08:55:15.352Z",
"file" => "/var/log/bbx1.log",
"host" => "s99",
"offset" => "0",
"type" => "disp"


(Magnus Bäck) #3

"fields": { "type": "disp" }

Shouldn't this be "bb_disp"?

I think you're overcomplicating things with the mutate filters. Just let the type field be the single condition that chooses which filters to apply.


(Lukas Meier) #4

first of all u are my hero magnusbaeck, from the last helps :smiley:

ah in my actual code i got the right one:
"fields": { "type": "bb_disp" }"

"Just let the type field be the single condition that chooses which filters to apply."

u mean in hole forwarder just 1 type?
but how can i chose then in logstash which filter is has to use?


(Magnus Bäck) #5

ah in my actual code i got the right one:
"fields": { "type": "bb_disp" }"

Okay. Are you not copy/pasting the configuration?

u mean in hole forwarder just 1 type?

No, keep the LSF configuration the same. Just drop the mutate filters. I don't understand what the point of them is. This should be enough:

filter {
  if [type] == "bb_disp" {
    grok {
      ...
    }
  } else if [type] = "..." {
    ...
  }
}

(Lukas Meier) #6

yes youre right it makes it easier thanks,

ok i think my problem lies on the grok patterns,
http://grokconstructor.appspot.com/ is very helpfull,
but on one "special part" of the parsing code the page crashes :confused:
when i want parse:
<#03>[30000] (09/29 00:00:00.123):{0x5F4} [DIAG] EVENT=Thread_report, THREADID=0xB0C, THREADNAME="UserWorkerThread"

all fine, but the "):{0x5F4}" part the page always crashes,
i could ):{%{BASE16FLOAT:dZahl}} but logstash didnt accept this,
and the page ALWAYS crashes when it comes to the "):{0x5F4}" part


(Lukas Meier) #7

ok im right the grok patterns for the second type arent right,
the 2 other types are fine --> only 1 right without grok error is saved
but my third type saves all types without any grok errors :frowning:


(Magnus Bäck) #8

To get effective help with grok issues, please post an example of a message that isn't correctly parsed together with your current configuration. If it's not completely obvious, include an example or description that shows what the desired output is.


(Lukas Meier) #9

this grok pattern:
"\A<#%{NUMBER:dRautenID}>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME}%{SYSLOG5424PRINTASCII:dUnparsable} [%{CISCO_REASON:dEventlevel}] %{CISCO_REASON:dEventJaNein}=%{CISCO_REASON:Eventmsg} %{GREEDYDATA:dMessage}"

should match on this log-line:
<#03>[30000] (09/29 00:00:00.123):{0x5F4} [DIAG] EVENT=Thread_report, THREADID=0xB0C, THREADNAME="UserWorkerThread"

but it parse all 3 types, without errors,
and the other type of logs:

2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG ** making TGT request .. **

<2015-09-29 14:04:14.142 MESZ>:[20102]:<MDS-CS_S15x>::<LAYER = IPPP, EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1>


(Lukas Meier) #10
  "message" => "2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG ** making TGT request .. **",
    "@version" => "1",
  "@timestamp" => "2015-10-01T12:33:54.671Z",
        "file" => "/var/log/bbx1.log",
        "host" => "s99",
      "offset" => "0",
        "type" => "bb_core",
      "cLevel" => "CORE",
     "cThread" => "wff-thread-98",
      "cQUEUE" => "loadDirGroupsUsersQueueConsumer",
       "cNone" => "none|utility",
"cCorrelation" => "[{}]",
     "cStatus" => "DEBUG",
    "cMessage" => "** making TGT request .. **"

}
{
"message" => "2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG ** making TGT request .. **",
"@version" => "1",
"@timestamp" => "2015-10-01T12:33:54.681Z",
"file" => "/var/log/bbx1.log",
"host" => "s99",
"offset" => "0",
"type" => "disp"
}
{
"message" => "<#03>[30000] (09/29 00:00:00.123):{0x5F4} [DIAG] EVENT=Thread_report, THREADID=0xAD8, THREADNAME="BIPPe_Controller"",
"@version" => "1",
"@timestamp" => "2015-10-01T12:33:54.682Z",
"file" => "/var/log/bbx1.log",
"host" => "s99",
"offset" => "140",
"type" => "disp"
}
{
"message" => "<2015-09-29 14:04:14.142 MESZ>:[20102]:<MDS-CS_S15.x.de>::<LAYER = IPPP, EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1>",
"@version" => "1",
"@timestamp" => "2015-10-01T12:33:54.683Z",
"file" => "/var/log/bbx1.log",
"host" => "s99",
"offset" => "256",
"type" => "disp"
}
{
"message" => "<2015-09-29 14:04:14.142 MESZ>:[20102]:<MDS-CS_S15.x.de>::<LAYER = IPPP, EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1>",
"@version" => "1",
"@timestamp" => "2015-10-01T12:33:54.686Z",
"file" => "/var/log/bbx1.log",
"host" => "s99",
"offset" => "256",
"type" => "bb_mdat",
"Zeitzone" => "MESZ",
"mID" => "20102",
"mDOMAIN" => "MDS-CS_S15x.de",
"mAction" => "DEBUG",
"mLayer" => "LAYER",
"mLayerinfo" => "IPPP",
"mMessage" => "EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1>"
}


(Magnus Bäck) #11

this grok pattern:
"\A<#%{NUMBER:dRautenID}>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME}%{SYSLOG5424PRINTASCII:dUnparsable} [%{CISCO_REASON:dEventlevel}] %{CISCO_REASON:dEventJaNein}=%{CISCO_REASON:Eventmsg} %{GREEDYDATA:dMessage}"

There are many problems with this grok expression, including missing escaping of parentheses and square brackets. The following works though:

<#%{NUMBER:dRautenID}>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}):{0x%{BASE16NUM}} [%{WORD:dEventlevel}] %{WORD:dEventJaNein}=%{WORD:Eventmsg}, %{GREEDYDATA:dMessage}


(Lukas Meier) #12

thanks for your help man,

but the test Homepage says:

not matched :frowning:


(Magnus Bäck) #13

"test Homepage"?


(Lukas Meier) #14

http://grokconstructor.appspot.com/

i think there is the problem:
<#%{NUMBER:dRautenID}>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}
this match till here,
but then comes the "):{0x5F4}" part, which thanks to you, you helped with your "):{0x%{BASE16NUM}} "
but copy paste dont help :frowning:

this of you not matching :frowning:
<#%{NUMBER:dRautenID}>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}):{0x%{BASE16NUM}} [%{WORD:dEventlevel}] %{WORD:dEventJaNein}=%{WORD:Eventmsg}, %{GREEDYDATA:dMessage}


(Magnus Bäck) #15

Argh, sorry. I forgot to format the expression properly so various backslashes were stripped. This should work better:

<#%{NUMBER:dRautenID}>\[%{NUMBER:dID}\] \(%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}\):{0x%{BASE16NUM}} \[%{WORD:dEventlevel}\] %{WORD:dEventJaNein}=%{WORD:Eventmsg}, %{GREEDYDATA:dMessage}

(Lukas Meier) #16

in shortform: man AWESOME thank you very much :smile:

god your my hero^^


(system) #17