# Convert values in kv-pairs

**URL:** <https://discuss.elastic.co/t/convert-values-in-kv-pairs/39904>\
**Category:** Logstash\
**Created:** [January 22, 2016, 4:40pm UTC](https://discuss.elastic.co/t/convert-values-in-kv-pairs/39904 "2016-01-22T16:40:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![intellion](https://avatars.discourse-cdn.com/v4/letter/i/f4b2a3/32.png) [@intellion](https://discuss.elastic.co/u/intellion)\
**Post date:** [January 22, 2016, 4:40pm UTC](https://discuss.elastic.co/t/convert-values-in-kv-pairs/39904/1 "2016-01-22T16:40:59Z")

</div>

Hii there,

I am trying to make some of our logs available in kibana using logstash-elasticsearch.  
the logs are not all structured. they do have some common patterns though.  
So far I could successfully extract date, logger, loglevel... this is where the similarities stop though and where i start having trouble.

Here is an example a loglines:

> - 22.01.2016 12:41:05.293\*; General.TcpPort.Data.CommunicationAssistance ; INFO ; 172.25.101.158:7211; Sending data; Request; Header; 0x5349; PayloadLength; 31; CommandId; 0x0032; CommandSequenceNumber; 1244; AntennaSelection; 1; RfChannel; 3; Retry; 0x10; CommunicationWindow; 2; CommunicationGroup; 0x75; ManId; 0x4C; RfProtocolTagType; 0x01; RfTagId; 411017529; MemoryType; Ram; Address; 0x00000A00; NumberBytes; 11; Data; 02-04-04-07-02-34-52-2C-00-48-5A; [CommandId; 2083322604]; [18]

Sometimes loglines contain key-value pairs, seperated by ';' sometime they don't.  
I already managed to extract the loglines with the kv-pairs in it using the following filter:

```
filter {
	# first start with the elements that are contained in each logline (like timestamp, logger, loglevel and message
	grok {
		# get logtimestampstring, logger, loglevel and the logmessage from the message tag
		match => { "message" => "%{DATESTAMP:logtimestampstring}\; (?<logger>([A-Za-z0-9]+([/.][A-Za-z0-9]+))+)%{SPACE}*(;) %{LOGLEVEL:loglevel} ; (?<message>(.+))" }
		overwrite => ["message"]
	}
	mutate {
            gsub => [
                "logtimestampstring", " ", ";"
                ]
    }
	# convert the timestampstring to a real date
	date {
            locale => "en"
            match => ["logtimestampstring", "dd.MM.YYYY;HH:mm:ss.SSS"]
            timezone => "Europe/Vienna"
            #target => "logtimestamp"
			target => "@timestamp"
			remove_field => ["logtimestampstring"]
    }
	grok {
		# check for any keyvaluemessages texts
		match => { "message" => "%{KEYVALUEMESSAGE:keyvaluemessage}" 
		break_on_match => false
		patterns_dir => [".patterns/lottrack"]}
		tag_on_failure => ["notkeyvalue"]
	}
	if "notkeyvalue" not in [tags] {	
		kv {
			source => "keyvaluemessage"
			trim => "[ms]"
			target => "kvpairs"
			value_split => ";"	
			field_split => ";"
		}
	}
}

```

this filter produces the the following output for the above logline:

```
{
            "message" => "172.25.101.158:7211; Sending data; Request; Header; 0x5349; PayloadLength; 31; CommandId; 0x0032; CommandSequenceNumber; 1244; AntennaSelection; 1; RfChannel; 3; Retry; 0x10; CommunicationWindow; 2; CommunicationGroup; 0x75; ManId; 0x4C; RfProtocolTagType; 0x01; RfTagId; 411017529; MemoryType; Ram; Address; 0x00000A00; NumberBytes; 11; Data; 02-04-04-07-02-34-52-2C-00-48-5A; [CommandId; 2083322604]; [18]\r",
           "@version" => "1",
         "@timestamp" => "2016-01-22T11:41:05.293Z",
               "host" => "DEVVS03",
             "logger" => "General.TcpPort.Data.CommunicationAssistance",
           "loglevel" => "INFO",
    "keyvaluemessage" => "; Header; 0x5349; PayloadLength; 31; CommandId; 0x0032; CommandSequenceNumber; 1244; AntennaSelection; 1; RfChannel; 3; Retry; 0x10; CommunicationWindow; 2; CommunicationGroup; 0x75; ManId; 0x4C; RfProtocolTagType; 0x01; RfTagId; 411017529",
            "kvpairs" => {
                       " Header" => "0x5349",
                " PayloadLength" => "31",
                    " CommandId" => "0x0032",
        " CommandSequenceNumber" => "1244",
             " AntennaSelection" => "1",
                    " RfChannel" => "3",
                        " Retry" => "0x10",
          " CommunicationWindow" => "2",
           " CommunicationGroup" => "0x75",
                        " ManId" => "0x4C",
            " RfProtocolTagType" => "0x01",
                      " RfTagId" => "411017529"
    }
}

```

Problem i have now is that i want to convert all the values of the kv-pairs that are either int or flot to these types.  
How do I do that?  
I would need something like iterating over all the keys in "kvpairs" and try to convert them in either float or int?  
So far i haven't found a way to do this though.

Thomas

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2016, 4:46pm UTC](https://discuss.elastic.co/t/convert-values-in-kv-pairs/39904/2 "2016-01-23T16:46:45Z")

</div>

You'll have to use a ruby filter. See [https://groups.google.com/d/msg/logstash-users/n1h8WVtAeJA/Svshm6uryHwJ](https://groups.google.com/d/msg/logstash-users/n1h8WVtAeJA/Svshm6uryHwJ) for an example that's close to what you need.

---

<div class="post-metadata">

**Author:** ![intellion](https://avatars.discourse-cdn.com/v4/letter/i/f4b2a3/32.png) [@intellion](https://discuss.elastic.co/u/intellion)\
**Post date:** [January 27, 2016, 9:41am UTC](https://discuss.elastic.co/t/convert-values-in-kv-pairs/39904/3 "2016-01-27T09:41:08Z")

</div>

thanks for the hint. i got it to work.  
this is how my ruby filter looks now:

> ` ruby{ code => " kv = event['kvpairs'] kv.to_hash.keys.each { |k| if kv[k].include? '.' kv[k] = kv[k].to_f else kv[k] = kv[k].to_i end } " }`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:14am UTC](https://discuss.elastic.co/t/convert-values-in-kv-pairs/39904/4 "2017-07-06T05:14:08Z")

</div>