# Logstash monitoring BB-Server Logs

**URL:** <https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457>\
**Category:** Logstash\
**Created:** [October 1, 2015, 9:11am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457 "2015-10-01T09:11:40Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 9:11am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/1 "2015-10-01T09:11:40Z")

</div>

hi my mission it is to monitor a Blackberry (BES) Server with logstash  
i got 3 types of Logfiles on BES-Server to parse,  
core, mdat and disp logfile-types,  
but they save wrong into ES,  
with 2 types it was more easier,  
but now with 3,  
i dont know how to solve the prob,  
someone can help pls?

logstash:  
input {  
lumberjack {  
# The port to listen on  
port =\> 5000

```
# The paths to your ssl cert and key
ssl_certificate => "/opt/logstash/etc/log.x.crt"
ssl_key => "/opt/logstash/etc/logstash.key"
    }

```

}

filter {  
if [type] == "bb\_core" {  
grok {  
match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601}%{SPACE}%{NOTSPACE}%{SPACE}%{PROG:cLevel} {%{SYSLOG5424PRINTASCII:cThread} : %{WORD:cQUEUE}} %{SYSLOG5424PRINTASCII:cNone} %{SYSLOG5424PRINTASCII:cCorrelation} - %{PROG:cStatus} %{GREEDYDATA:cMessage}" }

```
  			}

```

if "\_grokparsefailure" in [tags] {  
mutate {  
type =\> "bb\_disp" }  
}

```
	}

```

if [type] == "bb\_disp" {  
grok {  
match =\> { "message" =\> "\A\<#%{INT:dRautenID}\>[%{INT}] (%{NOTSPACE:dDate} %{HAPROXYTIME:dTIME}%{NOTSPACE:dID2} {%{JAVAFILE:dUsername}} %{CISCO\_REASON:dAction}(%{WORD}%{NOTSPACE} %{NOTSPACE:dID}%{SPACE}%{WORD}=%{NOTSPACE:dEmail} %{HOST}=%{WORD:dDeviceID};%{UUID:dCorrelation}, %{WORD}=%{WORD:dRouting}, %{USER}=\<%{WORD:dAgent}\>, %{USER}=%{WORD:dTime}, %{WORD}=%{INT:dExt}, %{WORD}=%{INT:dwl}%{NOTSPACE}%{SPACE}%{GREEDYDATA:dMessage}"}  
}

if "\_grokparsefailure" in [tags] {  
mutate {  
type =\> "bb\_mdat" }  
}  
}

if [type] == "bb\_mdat" {  
grok {  
match =\> { "message" =\> "\A\<%{TIMESTAMP\_ISO8601}%{SPACE}%{WORD}\>:[%{INT:mID}]:\<%{USERNAME:mDOMAIN}\>:\<%{WORD:mAction}\>:\<%{WORD:mLayer} = %{WORD:mLayerinfo}, %{GREEDYDATA:mMessage}"}  
}  
}

if "\_grokparsefailure" in [tags] {  
mutate {  
type =\> "bb\_core" }  
}

}

output {  
if "\_grokparsefailure" not in [tags] {  
elasticsearch { host =\> localhost }  
stdout { codec =\> rubydebug } } }

forwarder:

{  
"network": {  
"servers": ["log.x:5000"],  
"ssl ca": "/opt/logstash-forwarder/etc/log.x.crt",  
"timeout": 15  
},  
"files": [  
{  
"paths": [  
"/var/log/bbx1.log"  
],  
"fields": { "type": "bb\_core" }  
},

{  
"paths": [  
"/var/log/bbx1.log"  
],  
"fields": { "type": "disp" }  
},

{  
"paths": [  
"/var/log/bbx1.log"  
],  
"fields": { "type": "bb\_mdat" }  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 9:13am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/2 "2015-10-01T09:13:28Z")

</div>

i think the patterns are right,  
why its saving into wrong logtype?

"message" =\> "2015-09-29T11:47:08.876+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG ",  
"@version" =\> "1",  
"@timestamp" =\> "2015-10-01T08:55:15.350Z",  
"file" =\> "/var/log/bbx1.log",  
"host" =\> "s99",  
"offset" =\> "409",  
"type" =\> "bb\_core",  
"cLevel" =\> "CORE",  
"cThread" =\> "wff-thread-98",  
"cQUEUE" =\> "loadDirGroupsUsersQueueConsumer",  
"cNone" =\> "none|utility",  
"cCorrelation" =\> "[{}]",  
"cStatus" =\> "DEBUG"  
}  
{  
"message" =\> "2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG \*\* making TGT request .. \*\*",  
"@version" =\> "1",  
"@timestamp" =\> "2015-10-01T08:55:15.352Z",  
"file" =\> "/var/log/bbx1.log",  
"host" =\> "s99",  
"offset" =\> "0",  
"type" =\> "disp"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2015, 9:24am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/3 "2015-10-01T09:24:30Z")

</div>

> "fields": { "type": "disp" }

Shouldn't this be "bb\_disp"?

I think you're overcomplicating things with the mutate filters. Just let the `type` field be the single condition that chooses which filters to apply.

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 9:36am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/4 "2015-10-01T09:36:15Z")

</div>

first of all u are my hero magnusbaeck, from the last helps 😃

ah in my actual code i got the right one:  
"fields": { "type": "bb\_disp" }"

"Just let the type field be the single condition that chooses which filters to apply."

u mean in hole forwarder just 1 type?  
but how can i chose then in logstash which filter is has to use?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2015, 9:46am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/5 "2015-10-01T09:46:38Z")

</div>

> ah in my actual code i got the right one:  
> "fields": { "type": "bb\_disp" }"

Okay. Are you not copy/pasting the configuration?

> u mean in hole forwarder just 1 type?

No, keep the LSF configuration the same. Just drop the mutate filters. I don't understand what the point of them is. This should be enough:

```
filter {
  if [type] == "bb_disp" {
    grok {
      ...
    }
  } else if [type] = "..." {
    ...
  }
}

```

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 10:52am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/6 "2015-10-01T10:52:15Z")

</div>

yes youre right it makes it easier thanks,

ok i think my problem lies on the grok patterns,  
[http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/) is very helpfull,  
but on one "special part" of the parsing code the page crashes 😕  
when i want parse:  
\<#03\>[30000] (09/29 00:00:00.123):{0x5F4} [DIAG] EVENT=Thread\_report, THREADID=0xB0C, THREADNAME="UserWorkerThread"

all fine, but the "):{0x5F4}" part the page always crashes,  
i could ):{%{BASE16FLOAT:dZahl}} but logstash didnt accept this,  
and the page ALWAYS crashes when it comes to the "):{0x5F4}" part

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 11:12am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/7 "2015-10-01T11:12:51Z")

</div>

ok im right the grok patterns for the second type arent right,  
the 2 other types are fine --\> only 1 right without grok error is saved  
but my third type saves all types without any grok errors ☹

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2015, 11:41am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/8 "2015-10-01T11:41:26Z")

</div>

To get effective help with grok issues, please post an example of a message that isn't correctly parsed together with your current configuration. If it's not completely obvious, include an example or description that shows what the desired output is.

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 12:20pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/9 "2015-10-01T12:20:50Z")

</div>

this grok pattern:  
"\A\<#%{NUMBER:dRautenID}\>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME}%{SYSLOG5424PRINTASCII:dUnparsable} [%{CISCO\_REASON:dEventlevel}] %{CISCO\_REASON:dEventJaNein}=%{CISCO\_REASON:Eventmsg} %{GREEDYDATA:dMessage}"

should match on this log-line:  
\<#03\>[30000] (09/29 00:00:00.123):{0x5F4} [DIAG] EVENT=Thread\_report, THREADID=0xB0C, THREADNAME="UserWorkerThread"

but it parse all 3 types, without errors,  
and the other type of logs:

2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG \*\* making TGT request .. \*\*

\<2015-09-29 14:04:14.142 MESZ\>:[20102]:\<MDS-CS\_S15x\>::\<LAYER = IPPP, EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1\>

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 12:38pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/10 "2015-10-01T12:38:58Z")

</div>

```
  "message" => "2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG **making TGT request ..**",
    "@version" => "1",
  "@timestamp" => "2015-10-01T12:33:54.671Z",
        "file" => "/var/log/bbx1.log",
        "host" => "s99",
      "offset" => "0",
        "type" => "bb_core",
      "cLevel" => "CORE",
     "cThread" => "wff-thread-98",
      "cQUEUE" => "loadDirGroupsUsersQueueConsumer",
       "cNone" => "none|utility",
"cCorrelation" => "[{}]",
     "cStatus" => "DEBUG",
    "cMessage" => " **making TGT request ..**"

```

}  
{  
"message" =\> "2015-09-29T11:47:07.750+0200 - CORE {wff-thread-98 : loadDirGroupsUsersQueueConsumer} none|utility [{}] - DEBUG \*\* making TGT request .. \*\*",  
"@version" =\> "1",  
"@timestamp" =\> "2015-10-01T12:33:54.681Z",  
"file" =\> "/var/log/bbx1.log",  
"host" =\> "s99",  
"offset" =\> "0",  
"type" =\> "disp"  
}  
{  
"message" =\> "\<#03\>[30000] (09/29 00:00:00.123):{0x5F4} [DIAG] EVENT=Thread\_report, THREADID=0xAD8, THREADNAME="BIPPe\_Controller"",  
"@version" =\> "1",  
"@timestamp" =\> "2015-10-01T12:33:54.682Z",  
"file" =\> "/var/log/bbx1.log",  
"host" =\> "s99",  
"offset" =\> "140",  
"type" =\> "disp"  
}  
{  
"message" =\> "\<2015-09-29 14:04:14.142 MESZ\>:[20102]:\<MDS-CS\_S15.x.de\>::\<LAYER = IPPP, EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1\>",  
"@version" =\> "1",  
"@timestamp" =\> "2015-10-01T12:33:54.683Z",  
"file" =\> "/var/log/bbx1.log",  
"host" =\> "s99",  
"offset" =\> "256",  
"type" =\> "disp"  
}  
{  
"message" =\> "\<2015-09-29 14:04:14.142 MESZ\>:[20102]:\<MDS-CS\_S15.x.de\>::\<LAYER = IPPP, EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1\>",  
"@version" =\> "1",  
"@timestamp" =\> "2015-10-01T12:33:54.686Z",  
"file" =\> "/var/log/bbx1.log",  
"host" =\> "s99",  
"offset" =\> "256",  
"type" =\> "bb\_mdat",  
"Zeitzone" =\> "MESZ",  
"mID" =\> "20102",  
"mDOMAIN" =\> "MDS-CS\_S15x.de",  
"mAction" =\> "DEBUG",  
"mLayer" =\> "LAYER",  
"mLayerinfo" =\> "IPPP",  
"mMessage" =\> "EVENT = QueueSize, DEVICEPIN = 2bf90f7d, SendingQueueSize = 1\>"  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2015, 12:46pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/11 "2015-10-01T12:46:19Z")

</div>

> this grok pattern:  
> "\A\<#%{NUMBER:dRautenID}\>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME}%{SYSLOG5424PRINTASCII:dUnparsable} [%{CISCO\_REASON:dEventlevel}] %{CISCO\_REASON:dEventJaNein}=%{CISCO\_REASON:Eventmsg} %{GREEDYDATA:dMessage}"

There are many problems with this grok expression, including missing escaping of parentheses and square brackets. The following works though:

\<#%{NUMBER:dRautenID}\>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}):{0x%{BASE16NUM}} [%{WORD:dEventlevel}] %{WORD:dEventJaNein}=%{WORD:Eventmsg}, %{GREEDYDATA:dMessage}

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 1:35pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/12 "2015-10-01T13:35:37Z")

</div>

thanks for your help man,

but the test Homepage says:

not matched ☹

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2015, 1:36pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/13 "2015-10-01T13:36:49Z")

</div>

"test Homepage"?

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 1:40pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/14 "2015-10-01T13:40:04Z")

</div>

[http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/)

i think there is the problem:  
\<#%{NUMBER:dRautenID}\>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}  
this match till here,  
but then comes the "):{0x5F4}" part, which thanks to you, you helped with your "):{0x%{BASE16NUM}} "  
but copy paste dont help ☹

this of you not matching ☹  
\<#%{NUMBER:dRautenID}\>[%{NUMBER:dID}] (%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}):{0x%{BASE16NUM}} [%{WORD:dEventlevel}] %{WORD:dEventJaNein}=%{WORD:Eventmsg}, %{GREEDYDATA:dMessage}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2015, 1:44pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/15 "2015-10-01T13:44:14Z")

</div>

Argh, sorry. I forgot to format the expression properly so various backslashes were stripped. This should work better:

```
<#%{NUMBER:dRautenID}>\[%{NUMBER:dID}\] \(%{NUMBER:dDay}/%{NUMBER:dMonth} %{TIME:time}\):{0x%{BASE16NUM}} \[%{WORD:dEventlevel}\] %{WORD:dEventJaNein}=%{WORD:Eventmsg}, %{GREEDYDATA:dMessage}
```

---

<div class="post-metadata">

**Author:** ![lukas\_meier](https://avatars.discourse-cdn.com/v4/letter/l/df705f/32.png) [@lukas\_meier](https://discuss.elastic.co/u/lukas_meier)\
**Post date:** [October 1, 2015, 1:53pm UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/16 "2015-10-01T13:53:36Z")

</div>

in shortform: man AWESOME thank you very much 😄

god your my hero^^

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:27am UTC](https://discuss.elastic.co/t/logstash-monitoring-bb-server-logs/31457/17 "2017-07-06T05:27:33Z")

</div>