# How to process "Lat" & "Long" fields using default Logstash config and mapping to use in Kibana 4 tile map **URL:** \ **Category:** Elasticsearch\ **Created:** [April 25, 2015, 8:13pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398 "2015-04-25T20:13:45Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Rodger\_Moore](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rodger\_Moore](https://discuss.elastic.co/u/Rodger_Moore)\ **Post date:** [April 25, 2015, 8:13pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/1 "2015-04-25T20:13:45Z")
Hi there! My question is fairly simple but I'm having trouble finding a solution. I have a csv file containing Lat and Lon coordinates in separate fields named "Latitude" and "Longitude". Most of the info I found on the net is focussed on GeoIP (which is great functionality btw) but besides some posts [https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM](https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM)in Google Groups I failed finding a good tutorial for this use-case. What is the simplest way of getting separate Long / Lat fields into a geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the default Logstash (mapping) - ES - Kibana settings? I am using logstash 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. Summary: --\> csv containing Long / Lat in separate fields --\> Logstash --\> ES --\> Kibana4? Any help very much appreciated! Cheers, Rodger -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 25, 2015, 9:01pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/2 "2015-04-25T21:01:16Z")
You need to use the mutate filter and move your fields into a "location" one. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 25 avr. 2015 à 22:13, Rodger Moore [rodgerblom@gmail.com](mailto:rodgerblom@gmail.com) a écrit : > > Hi there! > > My question is fairly simple but I'm having trouble finding a solution. I have a csv file containing Lat and Lon coordinates in separate fields named "Latitude" and "Longitude". Most of the info I found on the net is focussed on GeoIP (which is great functionality btw) but besides some posts in Google Groups I failed finding a good tutorial for this use-case. > > What is the simplest way of getting separate Long / Lat fields into a geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the default Logstash (mapping) - ES - Kibana settings? I am using logstash 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash --\> ES --\> Kibana4? > > Any help very much appreciated! > > Cheers, > > ## Rodger > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3BF1A3A0-290C-48EA-84F1-58BB92BEAC2D%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/3BF1A3A0-290C-48EA-84F1-58BB92BEAC2D%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Rodger\_Moore](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rodger\_Moore](https://discuss.elastic.co/u/Rodger_Moore)\ **Post date:** [April 26, 2015, 9:17am UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/3 "2015-04-26T09:17:29Z")
Hi David, Thanks but I'm struggling getting this done. I tried different things. For example: ``` csv { columns => ["NAME","DATE","LAT","LONG"] separator => ";" add_tag => ["csv_parse_successfull"] } date { match => ["DATE", "dd-MM-YYYY HH:mm:ss"] } mutate { add_field => ["temp", "%{LAT}, %{LONG}"] } mutate { replace => ["geoip.location", "%{temp}"] } ``` } For some reason this converts the "geoip.location" type instead of keeping it a geo\_point type filled with ["%{LAT}, %{LONG}"]. Any suggestions? Thanks Op zaterdag 25 april 2015 23:01:29 UTC+2 schreef David Pilato: > You need to use the mutate filter and move your fields into a "location" > one. > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 25 avr. 2015 à 22:13, Rodger Moore \<[rodge...@gmail.com](mailto:rodge...@gmail.com) \\> > a écrit : > > Hi there! > > My question is fairly simple but I'm having trouble finding a solution. I > have a csv file containing Lat and Lon coordinates in separate fields named > "Latitude" and "Longitude". Most of the info I found on the net is focussed > on GeoIP (which is great functionality btw) but besides some posts > [https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM](https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM)in > Google Groups I failed finding a good tutorial for this use-case. > > What is the simplest way of getting separate Long / Lat fields into a > geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the > default Logstash (mapping) - ES - Kibana settings? I am using logstash > 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash --\> > ES --\> Kibana4? > > Any help very much appreciated! > > Cheers, > > Rodger > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/015a16b8-4379-42a6-8220-96be75600377%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/60d9811d-c960-40fe-9e61-dfb212c2602d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/60d9811d-c960-40fe-9e61-dfb212c2602d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Rodger\_Moore](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rodger\_Moore](https://discuss.elastic.co/u/Rodger_Moore)\ **Post date:** [April 26, 2015, 11:50am UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/4 "2015-04-26T11:50:54Z")
Hi there again! This problem is caused by, what I believe, a bug in Logstash or Elasticsearch. I used a very small test csv file with only 1 or 2 records per date. The default Logstash template creates 1 index per date. For some reason the creation of indices goes wrong when it comes to field types and very few records per index. After I changed the index creation template in the output config to: output { elasticsearch { protocol =\> "http" index =\> "logstash-%{+YYYY.MM}" } } thus creating only 1 index per month the problem with wrong field types was gone. If the folks from Elastic want to reproduce this, I enclosed the config files and test file. Changed status to solved. Cheers, Rodger. Op zaterdag 25 april 2015 22:13:45 UTC+2 schreef Rodger Moore: > Hi there! > > My question is fairly simple but I'm having trouble finding a solution. I > have a csv file containing Lat and Lon coordinates in separate fields named > "Latitude" and "Longitude". Most of the info I found on the net is focussed > on GeoIP (which is great functionality btw) but besides some posts > [https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM](https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM)in > Google Groups I failed finding a good tutorial for this use-case. > > What is the simplest way of getting separate Long / Lat fields into a > geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the > default Logstash (mapping) - ES - Kibana settings? I am using logstash > 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash --\> > ES --\> Kibana4? > > Any help very much appreciated! > > Cheers, > > Rodger -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/604b6de2-5c3f-4568-9f2a-8bf7e27611a2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/604b6de2-5c3f-4568-9f2a-8bf7e27611a2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 26, 2015, 4:34pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/5 "2015-04-26T16:34:01Z")
It's not an issue IMO but just a default configuration. FYI here is a sample config file I just used to parse some CSV data: input { stdin {} } filter { csv { separator =\> ";" columns =\> [ "id","name","slug","uic","uic8\_sncf","longitude","latitude", "parent\_station\_id","is\_city","country", "is\_main\_station","time\_zone","is\_suggestable","sncf\_id", "sncf\_is\_enabled","idtgv\_id","idtgv\_is\_enabled", "db\_id","db\_is\_enabled","idbus\_id","idbus\_is\_enabled","ouigo\_id", "ouigo\_is\_enabled", "trenitalia\_id","trenitalia\_is\_enabled","ntv\_id","ntv\_is\_enabled", "info\_fr", "info\_en","info\_de","info\_it","same\_as" ] } if [id] == "id" { drop { } } else { mutate { convert =\> { "longitude" =\> "float" } convert =\> { "latitude" =\> "float" } } ``` mutate { rename => { "longitude" => "[location][lon]" "latitude" => "[location][lat]" } } mutate { remove_field => ["message", "host", "@timestamp", "@version"] } ``` } } output { # stdout { codec =\> rubydebug } stdout { codec =\> dots } elasticsearch { protocol =\> "http" host =\> "localhost" index =\> "sncf" index\_type =\> "gare" template =\> "sncf\_template.json" template\_name =\> "sncf" document\_id =\> "%{id}" } } Hope this helps Le dimanche 26 avril 2015 13:50:54 UTC+2, Rodger Moore a écrit : > Hi there again! > > This problem is caused by, what I believe, a bug in Logstash or > Elasticsearch. I used a very small test csv file with only 1 or 2 records > per date. The default Logstash template creates 1 index per date. For some > reason the creation of indices goes wrong when it comes to field types and > very few records per index. After I changed the index creation template in > the output config to: > > output { > > elasticsearch { > protocol =\> "http" > index =\> "logstash-%{+YYYY.MM}" > } > } > > thus creating only 1 index per month the problem with wrong field types > was gone. If the folks from Elastic want to reproduce this, I enclosed the > config files and test file. Changed status to solved. > > Cheers, > > Rodger. > > Op zaterdag 25 april 2015 22:13:45 UTC+2 schreef Rodger Moore: > > > Hi there! > > > > My question is fairly simple but I'm having trouble finding a solution. I > > have a csv file containing Lat and Lon coordinates in separate fields named > > "Latitude" and "Longitude". Most of the info I found on the net is focussed > > on GeoIP (which is great functionality btw) but besides some posts > > [https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM](https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM)in > > Google Groups I failed finding a good tutorial for this use-case. > > > > What is the simplest way of getting separate Long / Lat fields into a > > geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the > > default Logstash (mapping) - ES - Kibana settings? I am using logstash > > 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash > > --\> ES --\> Kibana4? > > > > Any help very much appreciated! > > > > Cheers, > > > > Rodger -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/df9e85b7-35b8-4f3e-b361-fe8d2c33cac9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/df9e85b7-35b8-4f3e-b361-fe8d2c33cac9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Rodger\_Moore](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rodger\_Moore](https://discuss.elastic.co/u/Rodger_Moore)\ **Post date:** [April 28, 2015, 4:07pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/6 "2015-04-28T16:07:14Z")
Hi David, Thanks again for your answer. For some reason I am doing something wrong and its driving me nuts. I've tried your method but the tile map is showing me no results whatsoever. How did you define your template in Elasticsearch for this "location" field? Thanks, Rodger Op zondag 26 april 2015 18:34:01 UTC+2 schreef David Pilato: > It's not an issue IMO but just a default configuration. > > FYI here is a sample config file I just used to parse some CSV data: > > input { > stdin {} > } > > filter { > csv { > separator =\> ";" > columns =\> [ > "id","name","slug","uic","uic8\_sncf","longitude","latitude", > "parent\_station\_id","is\_city","country", > "is\_main\_station","time\_zone","is\_suggestable","sncf\_id", > "sncf\_is\_enabled","idtgv\_id","idtgv\_is\_enabled", > "db\_id","db\_is\_enabled","idbus\_id","idbus\_is\_enabled","ouigo\_id", > "ouigo\_is\_enabled", > "trenitalia\_id","trenitalia\_is\_enabled","ntv\_id","ntv\_is\_enabled", > "info\_fr", > "info\_en","info\_de","info\_it","same\_as" > ] > } > > if [id] == "id" { > drop { } > } else { > mutate { > convert =\> { "longitude" =\> "float" } > convert =\> { "latitude" =\> "float" } > } > > ``` > mutate { > rename => { > "longitude" => "[location][lon]" > "latitude" => "[location][lat]" > } > } > > mutate { > remove_field => ["message", "host", "@timestamp", "@version"] > } > > ``` > > } > } > > output { > > # stdout { codec =\> rubydebug } > > stdout { codec =\> dots } > elasticsearch { > protocol =\> "http" > host =\> "localhost" > index =\> "sncf" > index\_type =\> "gare" > template =\> "sncf\_template.json" > template\_name =\> "sncf" > document\_id =\> "%{id}" > } > } > > Hope this helps > > Le dimanche 26 avril 2015 13:50:54 UTC+2, Rodger Moore a écrit : > > > Hi there again! > > > > This problem is caused by, what I believe, a bug in Logstash or > > Elasticsearch. I used a very small test csv file with only 1 or 2 records > > per date. The default Logstash template creates 1 index per date. For some > > reason the creation of indices goes wrong when it comes to field types and > > very few records per index. After I changed the index creation template in > > the output config to: > > > > output { > > > > elasticsearch { > > protocol =\> "http" > > index =\> "logstash-%{+YYYY.MM}" > > } > > } > > > > thus creating only 1 index per month the problem with wrong field types > > was gone. If the folks from Elastic want to reproduce this, I enclosed the > > config files and test file. Changed status to solved. > > > > Cheers, > > > > Rodger. > > > > Op zaterdag 25 april 2015 22:13:45 UTC+2 schreef Rodger Moore: > > > > > Hi there! > > > > > > My question is fairly simple but I'm having trouble finding a solution. > > > I have a csv file containing Lat and Lon coordinates in separate fields > > > named "Latitude" and "Longitude". Most of the info I found on the net is > > > focussed on GeoIP (which is great functionality btw) but besides some > > > posts > > > [https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM](https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM)in > > > Google Groups I failed finding a good tutorial for this use-case. > > > > > > What is the simplest way of getting separate Long / Lat fields into a > > > geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the > > > default Logstash (mapping) - ES - Kibana settings? I am using logstash > > > 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > > > > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash > > > --\> ES --\> Kibana4? > > > > > > Any help very much appreciated! > > > > > > Cheers, > > > > > > Rodger -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 28, 2015, 6:44pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/7 "2015-04-28T18:44:54Z")
Lucky you! I just blogged about it 🙂 > **[Exploring Capitaine Train dataset - David Pilato](https://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/)** > > Recently I saw a tweet where Capitaine Train team started to open data they have collected and enriched or corrected. > Ouvrez, ouvrez, les données structurées. Capitaine Train libère les gares : https://t.co/y6DjWsbALF #opendata > — Trainline France... -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 28 avr. 2015 à 18:07, Rodger Moore [rodgerblom@gmail.com](mailto:rodgerblom@gmail.com) a écrit : > > Hi David, > > Thanks again for your answer. For some reason I am doing something wrong and its driving me nuts. I've tried your method but the tile map is showing me no results whatsoever. How did you define your template in Elasticsearch for this "location" field? > > Thanks, > > Rodger > > Op zondag 26 april 2015 18:34:01 UTC+2 schreef David Pilato: > > > It's not an issue IMO but just a default configuration. > > > > FYI here is a sample config file I just used to parse some CSV data: > > > > input { > > stdin {} > > } > > > > filter { > > csv { > > separator =\> ";" > > columns =\> [ > > "id","name","slug","uic","uic8\_sncf","longitude","latitude","parent\_station\_id","is\_city","country", > > "is\_main\_station","time\_zone","is\_suggestable","sncf\_id","sncf\_is\_enabled","idtgv\_id","idtgv\_is\_enabled", > > "db\_id","db\_is\_enabled","idbus\_id","idbus\_is\_enabled","ouigo\_id","ouigo\_is\_enabled", > > "trenitalia\_id","trenitalia\_is\_enabled","ntv\_id","ntv\_is\_enabled","info\_fr", > > "info\_en","info\_de","info\_it","same\_as" > > ] > > } > > > > if [id] == "id" { > > drop { } > > } else { > > mutate { > > convert =\> { "longitude" =\> "float" } > > convert =\> { "latitude" =\> "float" } > > } > > > > ``` > > mutate { > > rename => { > > "longitude" => "[location][lon]" > > "latitude" => "[location][lat]" > > } > > } > > > > mutate { > > remove_field => ["message", "host", "@timestamp", "@version"] > > } > > > > ``` > > > > } > > } > > > > output { > > > > # stdout { codec =\> rubydebug } > > > > stdout { codec =\> dots } > > elasticsearch { > > protocol =\> "http" > > host =\> "localhost" > > index =\> "sncf" > > index\_type =\> "gare" > > template =\> "sncf\_template.json" > > template\_name =\> "sncf" > > document\_id =\> "%{id}" > > } > > } > > > > Hope this helps > > > > Le dimanche 26 avril 2015 13:50:54 UTC+2, Rodger Moore a écrit : > > > > > Hi there again! > > > > > > This problem is caused by, what I believe, a bug in Logstash or Elasticsearch. I used a very small test csv file with only 1 or 2 records per date. The default Logstash template creates 1 index per date. For some reason the creation of indices goes wrong when it comes to field types and very few records per index. After I changed the index creation template in the output config to: > > > > > > output { > > > > > > elasticsearch { > > > protocol =\> "http" > > > index =\> "logstash-%{+YYYY.MM}" > > > } > > > } > > > > > > thus creating only 1 index per month the problem with wrong field types was gone. If the folks from Elastic want to reproduce this, I enclosed the config files and test file. Changed status to solved. > > > > > > Cheers, > > > > > > Rodger. > > > > > > Op zaterdag 25 april 2015 22:13:45 UTC+2 schreef Rodger Moore: > > > > > > > Hi there! > > > > > > > > My question is fairly simple but I'm having trouble finding a solution. I have a csv file containing Lat and Lon coordinates in separate fields named "Latitude" and "Longitude". Most of the info I found on the net is focussed on GeoIP (which is great functionality btw) but besides some posts in Google Groups I failed finding a good tutorial for this use-case. > > > > > > > > What is the simplest way of getting separate Long / Lat fields into a geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the default Logstash (mapping) - ES - Kibana settings? I am using logstash 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > > > > > > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash --\> ES --\> Kibana4? > > > > > > > > Any help very much appreciated! > > > > > > > > Cheers, > > > > > > > > Rodger > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9495D301-7A2E-47E5-A5E0-37D759E7AEFE%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/9495D301-7A2E-47E5-A5E0-37D759E7AEFE%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Rodger\_Moore](https://avatars.discourse-cdn.com/v4/letter/r/f9ae1b/32.png) [@Rodger\_Moore](https://discuss.elastic.co/u/Rodger_Moore)\ **Post date:** [April 28, 2015, 7:42pm UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/8 "2015-04-28T19:42:53Z")
Merci beaucoup, even in English! Nice blog, this helps a lot 🙂 Cheers Op dinsdag 28 april 2015 20:45:12 UTC+2 schreef David Pilato: > Lucky you! I just blogged about it 🙂 > > [https://david.pilato.fr/blog/2015-04-28-exploring-capitaine-train-dataset/](http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/) > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 28 avr. 2015 à 18:07, Rodger Moore \<[rodge...@gmail.com](mailto:rodge...@gmail.com) \\> > a écrit : > > Hi David, > > Thanks again for your answer. For some reason I am doing something wrong > and its driving me nuts. I've tried your method but the tile map is showing > me no results whatsoever. How did you define your template in Elasticsearch > for this "location" field? > > Thanks, > > Rodger > > Op zondag 26 april 2015 18:34:01 UTC+2 schreef David Pilato: > > > It's not an issue IMO but just a default configuration. > > > > FYI here is a sample config file I just used to parse some CSV data: > > > > input { > > stdin {} > > } > > > > filter { > > csv { > > separator =\> ";" > > columns =\> [ > > "id","name","slug","uic","uic8\_sncf","longitude","latitude", > > "parent\_station\_id","is\_city","country", > > "is\_main\_station","time\_zone","is\_suggestable","sncf\_id", > > "sncf\_is\_enabled","idtgv\_id","idtgv\_is\_enabled", > > "db\_id","db\_is\_enabled","idbus\_id","idbus\_is\_enabled","ouigo\_id", > > "ouigo\_is\_enabled", > > "trenitalia\_id","trenitalia\_is\_enabled","ntv\_id","ntv\_is\_enabled", > > "info\_fr", > > "info\_en","info\_de","info\_it","same\_as" > > ] > > } > > > > if [id] == "id" { > > drop { } > > } else { > > mutate { > > convert =\> { "longitude" =\> "float" } > > convert =\> { "latitude" =\> "float" } > > } > > > > ``` > > mutate { > > rename => { > > "longitude" => "[location][lon]" > > "latitude" => "[location][lat]" > > } > > } > > > > mutate { > > remove_field => ["message", "host", "@timestamp", "@version"] > > } > > > > ``` > > > > } > > } > > > > output { > > > > # stdout { codec =\> rubydebug } > > > > stdout { codec =\> dots } > > elasticsearch { > > protocol =\> "http" > > host =\> "localhost" > > index =\> "sncf" > > index\_type =\> "gare" > > template =\> "sncf\_template.json" > > template\_name =\> "sncf" > > document\_id =\> "%{id}" > > } > > } > > > > Hope this helps > > > > Le dimanche 26 avril 2015 13:50:54 UTC+2, Rodger Moore a écrit : > > > > > Hi there again! > > > > > > This problem is caused by, what I believe, a bug in Logstash or > > > Elasticsearch. I used a very small test csv file with only 1 or 2 records > > > per date. The default Logstash template creates 1 index per date. For some > > > reason the creation of indices goes wrong when it comes to field types and > > > very few records per index. After I changed the index creation template in > > > the output config to: > > > > > > output { > > > > > > elasticsearch { > > > protocol =\> "http" > > > index =\> "logstash-%{+YYYY.MM}" > > > } > > > } > > > > > > thus creating only 1 index per month the problem with wrong field types > > > was gone. If the folks from Elastic want to reproduce this, I enclosed the > > > config files and test file. Changed status to solved. > > > > > > Cheers, > > > > > > Rodger. > > > > > > Op zaterdag 25 april 2015 22:13:45 UTC+2 schreef Rodger Moore: > > > > > > > Hi there! > > > > > > > > My question is fairly simple but I'm having trouble finding a solution. > > > > I have a csv file containing Lat and Lon coordinates in separate fields > > > > named "Latitude" and "Longitude". Most of the info I found on the net is > > > > focussed on GeoIP (which is great functionality btw) but besides some > > > > posts > > > > [https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM](https://groups.google.com/forum/#!topic/elasticsearch/QaI1fj74RlM)in > > > > Google Groups I failed finding a good tutorial for this use-case. > > > > > > > > What is the simplest way of getting separate Long / Lat fields into a > > > > geo\_point and putting these coordinates on a Tile Map in Kibana 4 using the > > > > default Logstash (mapping) - ES - Kibana settings? I am using logstash > > > > 1.4.2 | Elasticsearch 1.5.0. and Kibana 4.0.1. > > > > > > > > Summary: --\> csv containing Long / Lat in separate fields --\> Logstash > > > > --\> ES --\> Kibana4? > > > > > > > > Any help very much appreciated! > > > > > > > > Cheers, > > > > > > > > Rodger > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/61c9e345-c997-43ac-ab58-7c753fecf0f0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1b0acfd1-442a-45a2-9a67-e3df3572b6f3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1b0acfd1-442a-45a2-9a67-e3df3572b6f3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:17am UTC](https://discuss.elastic.co/t/how-to-process-lat-long-fields-using-default-logstash-config-and-mapping-to-use-in-kibana-4-tile-map/23398/9 "2017-07-06T00:17:14Z")