Split a string with delimiter and assign to new key - logstash

Elastic Stack Logstash
1

I am trying to import csv into elasticsearch using logstash. I have a field in the csv with is an IP:Port pair. I would like to split the IP and port, and assign them to two new keys though the logstash config file.

I looked into kv , but kv seems to require the keys and values pair in the string. Mine is only value.

Sample csv data:

hostname1,172.20.28.11:8080,proxy_web,false
hostname2,172.20.28.20:443,web_https,false

In the above case, I already map the second field as "IP_Port" (using csv=> column) Now, I want to split 172.20.28.11:8080 as IP=172.20.28.11 and port=8080 (key=IP,value=172.20.28.11) and alike for port. Then mutate the datatype for the IP as 'ip datatype'.

After that, I can include the IP to geoip conversion using the following snippet, if I want to convert that extracted IP and use it to do geomapping.

geoip {
source => "Source IP"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
Any suggestions on this?

2

Use a csv filter to parse the record and get the ip and port into a single field. Then run another csv filter on the ip field, but with : used as a separator. You could use a grok or dissect filter as well.

3

You don't need the add_field directives. The geoip plugin does all that for you. I would use a mapping rather than setting the field as a float with mutate.

4

Thankyou for the suggestions All.
I found the grok option gave me precisely what I wanted. I could extract the IP (v4 or v6) using the below. I realized I dont need the port into a field, so this one worked realy faster in terms of processing as well.

filter {
grok {
match => { "message" => "%{IP:peer_ip}" }
}
}

5

Thankyou Aaron. Figured it out the output mapping already had geo_point mapping. So it worked out well.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.