Graph based on a Value within a String

Elastic Stack Kibana
1

Hello Community. I am fairly new to Kibana but reading the manuals I got it to a good state so far.

I am having a problem which is as follows:

I have ELK working and in Kibana I can go to discover and find the following records:

Table
JSON
@timestamp July 30th 2016, 16:54:50.000
t@version 1
t_id AVY8hAU1XLw-TOW0T-WC
t_index logstash-2016.07.30
#_score
t_type rpmstats
#average_rtt 523
tdevice_name MX80-6
#facility 3
tfacility_label system
thost 172.25.90.48
#jitter 0
#loss_percent 0
tlsp_name EF1-32158_mx80-6-mx240-2
#max_rtt 523
tmessage RPM_TEST_RESULTS: test-owner=northstar-lsp test-name=EF1-32158_mx80-6-mx240-2 loss=0.000000 min-rtt=523 max-rtt=523 avgerage-rtt=523 jitter=0
#min_rtt 523
#priority 30
tprogram cscript
#severity 6
tseverity_label Informational
ttags lsp
ttest_owner northstar-lsp
ttimestamp Jul 30 11:54:50
ttype rpmstats

As you can see above I have a field called 'message' and in that field I have a "loss=0.000000" ... I would like to create a line chart based on that 'loss' value so that I can graph and track the changed on that ...

Can you help ?

Thanks

Nuno Ferreira

2

You need to extract the field from the message, KB cannot do that for you.
Using Logstash will do this for you.

3

Can you give me an example please ?

4

Your best bet is to read over https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html and then come back with any other questions :slight_smile:

5

Mark, this is my configuration file for this bit is like this:

cat /etc/logstash/conf.d/rpm.conf
input {
syslog {
port => 1514
type => 'rpmstats'
}
}

filter {
if ([type] == 'rpmstats') {
grok {
match => {
"message" => "RPM_TEST_RESULTS: test-owner=%{USER:test_owner} test-name=%{NOTSPACE:test_name} loss=%{NUMBER:loss_percent} min-rtt=%{NUMBER:min_rtt} max-rtt=%{NUMBER:max_rtt} avgerage-rtt=%{NUMBER:average_rtt} jitter=%{NUMBER:jitter}"
}
}
mutate {
convert => {
"loss_percent" => "float"
"average_rtt" => "float"
"min_rtt" => "float"
"max_rtt" => "float"
"jitter" => "float"
}
rename => { "logsource" => "device_name" }
}
}

if ([test_owner] == 'northstar-lsp') {
	mutate {
	    rename => { "test_name" => "lsp_name" }
	    add_tag => [ "lsp" ]
	}
} else if ([test_owner] == 'northstar-ifl') {
	mutate {
		rename => { "test_name" => "interface_name" }
	    add_tag => [ "ifl" ]
	}
}

}

output {
if ([type] == 'rpmstats') {
elasticsearch {
hosts => [ "localhost:9200" ]
document_type => "rpmstats"
}

stdout {

codec => rubydebug { metadata => "true" }

}

}

}

I wonder if I have to do some different mutation to get the loss in a specific field that I can then graph on ?

Thanks a lot for your help ..

Nuno

6

Gurus,

Based on the above conf, shouldn't I be getting the value loss into the variable loss_percent ?

From what I understand that should be the case. but if the loss comes up as 100% then I get no loss_percent at all ...

Can someone help please ?

Thanks

7

What does the stdout show?

8

In Kibana this is what I'm getting:

September 1st 2016, 15:22:22.000 message:RPM_TEST_RESULTS: test-owner=northstar-lsp test-name=AF2-42156_mx80-4-mx240-2 loss=0.000000 min-rtt=1215 max-rtt=1780 avgerage-rtt=1422 jitter=565 @version:1 @timestamp:September 1st 2016, 15:22:22.000 type:rpmstats host:172.25.90.46 priority:30 timestamp:Sep 1 10:22:22 program:cscript severity:6 facility:3 facility_label:system severity_label:Informational test_owner:northstar-lsp loss_percent:0 min_rtt:1,215 max_rtt:1,780 average_rtt:1,422 jitter:565 device_name:MX80-4 lsp_name:AF2-42156_mx80-4-mx240-2 tags:lsp _id:AVbmIZxuu-rzbV2JZY2Q _type:rpmstats _index:logstash-2016.09.01 _score:

if you see abote the loss is 0 so I get the loss_percent as 0 as well

But below you can see that I have the loss at 100% and the loss_percent thing just disappears ...

September 1st 2016, 15:22:27.000 message:RPM_TEST_RESULTS: test-owner=northstar-lsp test-name=AF30-1236_br1dc1 loss=100.00000 min-rtt=NaN max-rtt=NaN avgerage-rtt=NaN jitter=NaN @version:1 @timestamp:September 1st 2016, 15:22:27.000 type:rpmstats host:172.25.90.42 priority:30 timestamp:Sep 1 10:22:27 program:cscript severity:6 facility:3 facility_label:system severity_label:Informational tags:_grokparsefailure device_name:MX240-2 _id:AVbmIa3Xu-rzbV2JZY3S _type:rpmstats _index:logstash-2016.09.01 _score:

9

Anyone can shed a light please ?

Thanks

10

To add to the above request I have noticed that when the loss=100.00000 I get a tags:grokparsefailure (as per above message)

I have run it through the parser and it doesn't give me any error ... but obviously it is in the real thing

I believe is due to some of the variables having a value of "NaN" when they are defined as a float.

Can someone help me in hat changes I need to make to have the grok accepting those values ?

Thanks

11

OK I think I managed to get this working :slight_smile:

I changed the message input to WORD and then convert it properly - This allows the grok not to fail which puts the variables in the right place and get things working

Cheers