I'm trying to parse data from a CSV cell so that I can use the latitude and longitude present in the cell for a geo_point visualization. As of right now, the column is titled Location and the cells after that have the location listed with an address and then the latitude and longitude of the school within parentheses.
I've been reading through the filters page to see what my best option might be but between grok, dissect, split or extract numbers I'm not entirely sure which filter is the most efficient way to pull out the lat and long.
I've read through a previous post, this ones a good start, so I'm working off of that info to try and create a visualization. The data that I'm working with is from NYS School Lead Testing and my overall goal is to learn how to import the data and create visualizations from it. I've got the import part down and some of the visualization parts setup but its not perfect so I'm trying to figure out how to manipulate to what we need. For this project I've stripped the whole thing down to just the Location column to try and work out how to parse it correctly.
Learning how to setup the mapping for this will obviously be a step thats needed at the end.
From reading the
CSV column title and data example
Location
7 CLEVELAND DR
ADDISON, NY 14801
(42.102851749, -77.231278965)
Current Logstash config called removeLatLong
input {
file {
path => "/tmp/removeLatLong.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
tags => "removeLatLong"
}
}
filter {
csv {
separator => ","
columns => [ "Location" ]
}
mutate {
convert => { "Location" => "string" }
convert => { "Latitude" => "float" }
convert => { "Longitude" => "float" }
}
mutate {
rename => {
"Latitude" => "[location][lat]"
"Longitude" => "[location][lon]"
}
}
#date {
#match => [ "Latest Record Update", "ISO8601" ]
#target => "Latest Record Update"
#}
}
output {
elasticsearch {
hosts => ["https://abc.ny.gov:9200", "https://def.ny.gov:9200", "https://ghi.ny.gov:9200"]
manage_template => false
index => "removeLatLong"
user => "logstash_internal"
password => "#####"
ssl => true
#ssl_certificate_verification => false
cacert => "/etc/logstash/globalcert/ca/ca.crt"
}
}
OK, so that grok still works