Graph based on a Value within a String

Hello Community. I am fairly new to Kibana but reading the manuals I got it to a good state so far.

I am having a problem which is as follows:

I have ELK working and in Kibana I can go to discover and find the following records:

@timestamp July 30th 2016, 16:54:50.000
t@version 1
t_index logstash-2016.07.30
t_type rpmstats
#average_rtt 523
tdevice_name MX80-6
#facility 3
tfacility_label system
#jitter 0
#loss_percent 0
tlsp_name EF1-32158_mx80-6-mx240-2
#max_rtt 523
tmessage RPM_TEST_RESULTS: test-owner=northstar-lsp test-name=EF1-32158_mx80-6-mx240-2 loss=0.000000 min-rtt=523 max-rtt=523 avgerage-rtt=523 jitter=0
#min_rtt 523
#priority 30
tprogram cscript
#severity 6
tseverity_label Informational
ttags lsp
ttest_owner northstar-lsp
ttimestamp Jul 30 11:54:50
ttype rpmstats

As you can see above I have a field called 'message' and in that field I have a "loss=0.000000" ... I would like to create a line chart based on that 'loss' value so that I can graph and track the changed on that ...

Can you help ?


Nuno Ferreira

You need to extract the field from the message, KB cannot do that for you.
Using Logstash will do this for you.

Can you give me an example please ?

Your best bet is to read over and then come back with any other questions :slight_smile:

Mark, this is my configuration file for this bit is like this:

cat /etc/logstash/conf.d/rpm.conf
input {
syslog {
port => 1514
type => 'rpmstats'

filter {
if ([type] == 'rpmstats') {
grok {
match => {
"message" => "RPM_TEST_RESULTS: test-owner=%{USER:test_owner} test-name=%{NOTSPACE:test_name} loss=%{NUMBER:loss_percent} min-rtt=%{NUMBER:min_rtt} max-rtt=%{NUMBER:max_rtt} avgerage-rtt=%{NUMBER:average_rtt} jitter=%{NUMBER:jitter}"
mutate {
convert => {
"loss_percent" => "float"
"average_rtt" => "float"
"min_rtt" => "float"
"max_rtt" => "float"
"jitter" => "float"
rename => { "logsource" => "device_name" }

if ([test_owner] == 'northstar-lsp') {
	mutate {
	    rename => { "test_name" => "lsp_name" }
	    add_tag => [ "lsp" ]
} else if ([test_owner] == 'northstar-ifl') {
	mutate {
		rename => { "test_name" => "interface_name" }
	    add_tag => [ "ifl" ]


output {
if ([type] == 'rpmstats') {
elasticsearch {
hosts => [ "localhost:9200" ]
document_type => "rpmstats"

stdout {

codec => rubydebug { metadata => "true" }




I wonder if I have to do some different mutation to get the loss in a specific field that I can then graph on ?

Thanks a lot for your help ..



Based on the above conf, shouldn't I be getting the value loss into the variable loss_percent ?

From what I understand that should be the case. but if the loss comes up as 100% then I get no loss_percent at all ...

Can someone help please ?


What does the stdout show?

In Kibana this is what I'm getting:

September 1st 2016, 15:22:22.000 message:RPM_TEST_RESULTS: test-owner=northstar-lsp test-name=AF2-42156_mx80-4-mx240-2 loss=0.000000 min-rtt=1215 max-rtt=1780 avgerage-rtt=1422 jitter=565 @version:1 @timestamp:September 1st 2016, 15:22:22.000 type:rpmstats host: priority:30 timestamp:Sep 1 10:22:22 program:cscript severity:6 facility:3 facility_label:system severity_label:Informational test_owner:northstar-lsp loss_percent:0 min_rtt:1,215 max_rtt:1,780 average_rtt:1,422 jitter:565 device_name:MX80-4 lsp_name:AF2-42156_mx80-4-mx240-2 tags:lsp _id:AVbmIZxuu-rzbV2JZY2Q _type:rpmstats _index:logstash-2016.09.01 _score:

if you see abote the loss is 0 so I get the loss_percent as 0 as well

But below you can see that I have the loss at 100% and the loss_percent thing just disappears ...

September 1st 2016, 15:22:27.000 message:RPM_TEST_RESULTS: test-owner=northstar-lsp test-name=AF30-1236_br1dc1 loss=100.00000 min-rtt=NaN max-rtt=NaN avgerage-rtt=NaN jitter=NaN @version:1 @timestamp:September 1st 2016, 15:22:27.000 type:rpmstats host: priority:30 timestamp:Sep 1 10:22:27 program:cscript severity:6 facility:3 facility_label:system severity_label:Informational tags:_grokparsefailure device_name:MX240-2 _id:AVbmIa3Xu-rzbV2JZY3S _type:rpmstats _index:logstash-2016.09.01 _score:

Anyone can shed a light please ?


To add to the above request I have noticed that when the loss=100.00000 I get a tags:grokparsefailure (as per above message)

I have run it through the parser and it doesn't give me any error ... but obviously it is in the real thing

I believe is due to some of the variables having a value of "NaN" when they are defined as a float.

Can someone help me in hat changes I need to make to have the grok accepting those values ?


OK I think I managed to get this working :slight_smile:

I changed the message input to WORD and then convert it properly - This allows the grok not to fail which puts the variables in the right place and get things working