KV-Pairs and type conversion

[desete]

Hello,

I have an input like this

<14>1 2017-09-11T14:00:27.582+02:00 SECRETHOSTNAME RT_FLOW - RT_FLOW_SESSION_CREATE [junos@2636.1.1.1.2.40 source-address="1.1.1.1" source-port="53780" destination-address="2.2.2.2" destination-port="389" service-name="junos-ldap" nat-source-address="3.3.3.3" nat-source-port="53780" nat-destination-address="4.4.4.4" nat-destination-port="389" src-nat-rule-type="N/A" src-nat-rule-name="N/A" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="6" policy-name="newNetAccess" source-zone-name="BLUB" destination-zone-name="BLAA" session-id-32="28173" username="N/A" roles="N/A" packet-incoming-interface="reth0.999" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"]

I managed to extract all fields I need with kv-pairs filter.
But... everything seems to be a string, so I cannot draw and things like that as ip-addresses are not ip-addresses.
I searched a lot now, but I can't find a solution.

this is what I am doing now and my question is how can I determine for each pair what type of data it is?

if "RT_FLOW" in [message] {
        grok {
        match => [ "message","%{GREEDYDATA:kvpairs}" ]
  }

kv {
   source => "kvpairs"
}

thank you very much in advance

//desete

[magnusbaeck (Magnus Bäck)]

But... everything seems to be a string, so I cannot draw and things like that as ip-addresses are not ip-addresses.

Whether a field is mapped as an IP address depends on the mappings of the ES index. You can use index templates to explicitly map fields as e.g. IP addresses.

You can use a mutate filter and its convert option to convert string fields in Logstash to numbers but even though you have fields containing numbers (like ports) they're not numbers that you'd typically process numerically (like compute sums or averages) so it's not terribly important.

[desete]

thanks Magnus.

You can use index templates to explicitly map fields as e.g. IP addresses.

any recommended reading?

thx and br
//seb

[kmsasidhar (SasidharReddy)]

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html