Convert values in kv-pairs

Elastic Stack Logstash
1

Hii there,

I am trying to make some of our logs available in kibana using logstash-elasticsearch.
the logs are not all structured. they do have some common patterns though.
So far I could successfully extract date, logger, loglevel... this is where the similarities stop though and where i start having trouble.

Here is an example a loglines:

  • 22.01.2016 12:41:05.293*; General.TcpPort.Data.CommunicationAssistance ; INFO ; 172.25.101.158:7211; Sending data; Request; Header; 0x5349; PayloadLength; 31; CommandId; 0x0032; CommandSequenceNumber; 1244; AntennaSelection; 1; RfChannel; 3; Retry; 0x10; CommunicationWindow; 2; CommunicationGroup; 0x75; ManId; 0x4C; RfProtocolTagType; 0x01; RfTagId; 411017529; MemoryType; Ram; Address; 0x00000A00; NumberBytes; 11; Data; 02-04-04-07-02-34-52-2C-00-48-5A; [CommandId; 2083322604]; [18]

Sometimes loglines contain key-value pairs, seperated by ';' sometime they don't.
I already managed to extract the loglines with the kv-pairs in it using the following filter:

filter {
	# first start with the elements that are contained in each logline (like timestamp, logger, loglevel and message
	grok {
		# get logtimestampstring, logger, loglevel and the logmessage from the message tag
		match => { "message" => "%{DATESTAMP:logtimestampstring}\; (?<logger>([A-Za-z0-9]+([/.][A-Za-z0-9]+))+)%{SPACE}*(;) %{LOGLEVEL:loglevel} ; (?<message>(.+))" }
		overwrite => [ "message" ]
	}
	mutate {
            gsub => [
                "logtimestampstring", " ", ";"
                ]
    }
	# convert the timestampstring to a real date
	date {
            locale => "en"
            match => ["logtimestampstring", "dd.MM.YYYY;HH:mm:ss.SSS"]
            timezone => "Europe/Vienna"
            #target => "logtimestamp"
			target => "@timestamp"
			remove_field => ["logtimestampstring"]
    }
	grok {
		# check for any keyvaluemessages texts
		match => { "message" => "%{KEYVALUEMESSAGE:keyvaluemessage}" 
		break_on_match => false
		patterns_dir => [".patterns/lottrack"]}
		tag_on_failure => [ "notkeyvalue" ]
	}
	if "notkeyvalue" not in [tags] {	
		kv {
			source => "keyvaluemessage"
			trim => "[ms]"
			target => "kvpairs"
			value_split => ";"	
			field_split => ";"
		}
	}
}

this filter produces the the following output for the above logline:

{
            "message" => "172.25.101.158:7211; Sending data; Request; Header; 0x5349; PayloadLength; 31; CommandId; 0x0032; CommandSequenceNumber; 1244; AntennaSelection; 1; RfChannel; 3; Retry; 0x10; CommunicationWindow; 2; CommunicationGroup; 0x75; ManId; 0x4C; RfProtocolTagType; 0x01; RfTagId; 411017529; MemoryType; Ram; Address; 0x00000A00; NumberBytes; 11; Data; 02-04-04-07-02-34-52-2C-00-48-5A; [CommandId; 2083322604]; [18]\r",
           "@version" => "1",
         "@timestamp" => "2016-01-22T11:41:05.293Z",
               "host" => "DEVVS03",
             "logger" => "General.TcpPort.Data.CommunicationAssistance",
           "loglevel" => "INFO",
    "keyvaluemessage" => "; Header; 0x5349; PayloadLength; 31; CommandId; 0x0032; CommandSequenceNumber; 1244; AntennaSelection; 1; RfChannel; 3; Retry; 0x10; CommunicationWindow; 2; CommunicationGroup; 0x75; ManId; 0x4C; RfProtocolTagType; 0x01; RfTagId; 411017529",
            "kvpairs" => {
                       " Header" => "0x5349",
                " PayloadLength" => "31",
                    " CommandId" => "0x0032",
        " CommandSequenceNumber" => "1244",
             " AntennaSelection" => "1",
                    " RfChannel" => "3",
                        " Retry" => "0x10",
          " CommunicationWindow" => "2",
           " CommunicationGroup" => "0x75",
                        " ManId" => "0x4C",
            " RfProtocolTagType" => "0x01",
                      " RfTagId" => "411017529"
    }
}

Problem i have now is that i want to convert all the values of the kv-pairs that are either int or flot to these types.
How do I do that?
I would need something like iterating over all the keys in "kvpairs" and try to convert them in either float or int?
So far i haven't found a way to do this though.

Thomas

2

You'll have to use a ruby filter. See https://groups.google.com/d/msg/logstash-users/n1h8WVtAeJA/Svshm6uryHwJ for an example that's close to what you need.

3

thanks for the hint. i got it to work.
this is how my ruby filter looks now:

ruby{ code => " kv = event['kvpairs'] kv.to_hash.keys.each { |k| if kv[k].include? '.' kv[k] = kv[k].to_f else kv[k] = kv[k].to_i end } " }