# Insert longitude and latitude to logstash config file

**URL:** <https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615>\
**Category:** Logstash\
**Created:** [November 14, 2017, 7:51pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615 "2017-11-14T19:51:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 14, 2017, 7:51pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/1 "2017-11-14T19:51:13Z")

</div>

Hello,  
I have CSV file with two geolocation columns:

> cell\_easting cell\_northing  
> 26.1541 66.48703  
> 26.161 66.49312  
> 26.166 66.49182

I would like to insert them as a markers into a map  
Here my config file:

```
> input {
> file {
> path => "/home/ahmed/Desktop/tfJuni_mini.csv"
> start_position => "beginning"
> sincedb_path => "/dev/null"
> }
> }
> filter {
> csv {
> separator => ","
> columns => ["cell_easting", "cell_northing", "subsperbase", "date_trunc"] 
> }
> date { match => ["date_trunc", "dd.MM.yyyy HH:mm:ss"] }
> mutate {convert => ["subsperbase", "integer"] }
> 
> if [cell_easting] and [cell_northing] {
> mutate {
> add_field => { "location" => "%{cell_northing}" }
> add_field => { "location" => "%{cell_easting}" }
> }
> mutate { convert => ["[location]", "geo_point" ] }
> }
> }
> 
> output {
> elasticsearch {
> hosts => ["http://elastic:changeme@127.0.0.1:9200"]
> index => "tfjuni"
> document_type => "tfJuni"
> }
> stdout {}
> }

```

Here is the command line to execute:  
`sudo /usr/share/logstash/bin/logstash --path.settings=/etc/logstash/ -f /home/ahmed/Desktop/TF.conf`

The prob is that I don't find my index ?  
any problem with the config file ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 14, 2017, 7:52pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/2 "2017-11-14T19:52:35Z")

</div>

If Logstash has problems sending data to Elasticsearch you'll typically find clues in the logs.

I'm not sure you can list the username and password in the ES URL.

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 14, 2017, 7:56pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/3 "2017-11-14T19:56:38Z")

</div>

Here is the log of : **/var/log/logstash/logstash-plain.log**

```
017-11-14T20:38:14,592][ERROR][logstash.agent] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: translation missing: en.logstash.agent.configuration.invalid_plugin_register>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.1.6/lib/logstash/filters/mutate.rb:190:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-mutate-3.1.6/lib/logstash/filters/mutate.rb:184:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:in `start_pipeline'"]}
[2017-11-14T20:38:14,697][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2017-11-14T20:38:17,654][WARN][logstash.agent] stopping pipeline {:id=>".monitoring-logstash"}
[2017-11-14T20:38:19,237][WARN][logstash.agent] stopping pipeline {:id=>"main"}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 14, 2017, 8:11pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/4 "2017-11-14T20:11:32Z")

</div>

> ```
> mutate { convert => ["[location]", "geo_point" ] }
> 
> ```

As documented, geo\_point is not a valid type for conversions. Just a few minutes ago I responded to another question related to geo\_point. That answer should be useful to you too.

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 14, 2017, 8:15pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/5 "2017-11-14T20:15:13Z")

</div>

I have changed the conf file like this, but the same prob:

> input {  
> file {  
> path =\> "/home/ahmed/Desktop/tfJuni\_mini.csv"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> }  
> }  
> filter {  
> csv {  
> separator =\> ","  
> columns =\> ["cell\_easting", "cell\_northing", "subsperbase", "date\_trunc"]  
> }  
> date { match =\> ["date\_trunc", "dd.MM.yyyy HH:mm:ss"] }  
> mutate {convert =\> ["subsperbase", "integer"] }  
> mutate { convert =\> ["[cell\_easting]", "geo\_point" ] }  
> mutate { convert =\> ["[cell\_northing]", "geo\_point" ] }
> 
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["[http://elastic:changeme@127.0.0.1:9200](http://elastic:changeme@127.0.0.1:9200)"]  
> index =\> "tfjuni"  
> document\_type =\> "tfJuni"  
> }  
> stdout {}  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 14, 2017, 9:58pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/6 "2017-11-14T21:58:46Z")

</div>

> mutate { convert =\> ["[cell\_easting]", "geo\_point" ] }  
> mutate { convert =\> ["[cell\_northing]", "geo\_point" ] }

Again, this won't work. geo\_point is not a valid conversion type for the mutate filter.

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 14, 2017, 10:13pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/7 "2017-11-14T22:13:15Z")

</div>

How can I read these values as a geo-location value ?  
Is it possible as a float type ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 15, 2017, 6:59am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/8 "2017-11-15T06:59:54Z")

</div>

So dId you read the other post I referred you to?

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 15, 2017, 8:50am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/9 "2017-11-15T08:50:20Z")

</div>

YES I have read it, the prob is to convert from csv columns to geo\_ip

> **[GeoIP in the Elastic Stack - Elasticsearch, Logstash, Ingest API](https://www.elastic.co/blog/geoip-in-the-elastic-stack)**
>
> Discover what browsers access your site. See where your end users are logging in from. Put your IP addresses or hostnames to work with geoip filtering.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 15, 2017, 12:58pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/10 "2017-11-15T12:58:22Z")

</div>

I've continued the old topic ([Create geopoint data](https://discuss.elastic.co/t/create-geopoint-data/107580/5)). You're pretty much asking the same thing so please follow that topic.

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 15, 2017, 1:34pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/11 "2017-11-15T13:34:37Z")

</div>

OK here is my problem,  
I have add a new field, look to my config file:

> input {  
> file {  
> path =\> "/home/ahmed/Desktop/tfJuni\_mini.csv"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> }  
> }  
> filter {  
> csv {  
> separator =\> ","  
> columns =\> ["cell\_easting", "cell\_northing", "subsperbase", "date\_trunc"]  
> }  
> mutate {convert =\> ["subsperbase", "integer"] }  
> mutate {  
> add\_field =\> ["[geoip][location]","%{cell\_easting}" ]  
> add\_field =\> ["[geoip][location]","%{cell\_northing}" ]  
> }  
> mutate {convert =\> {"[geoip][location]" =\> "float"} }
> 
> }
> 
> output {  
> elasticsearch {  
> hosts =\> ["[http://elastic:changeme@127.0.0.1:9200](http://elastic:changeme@127.0.0.1:9200)"]  
> index =\> "tfjuni"  
> document\_type =\> "tfJuni"  
> }  
> stdout { codec =\> rubydebug }  
> }

In kibana the data shows fine,

 ![Screenshot-2017-11-15 Kibana](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60ce213dfb664ba63c25d44c31d5a6abb118b6ba.png)

But in the visualization section there is an error:

 ![Screenshot-2017-11-15 Kibana(1)](https://us1.discourse-cdn.com/elastic/original/3X/5/3/530d8f1b00381cdf7d5750b73ad408a197e38cd7.png)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 15, 2017, 1:44pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/12 "2017-11-15T13:44:27Z")

</div>

You're not following the advice I just gave in the other topic. Pay attention to the last paragraph.

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 17, 2017, 10:00am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/13 "2017-11-17T10:00:38Z")

</div>

Tanks for your replay, I have add a template to mapping the field

```
curl -XPUT 'localhost:9200/_template/tfjuni?pretty' -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["tfjuni"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "geo.location": {
          "type": "geo_point"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z YYYY"
        }
      }
    }
  }
}

```

I got this as error:

> ```
> {
> "error" : {
> "root_cause" : [
> {
> "type" : "action_request_validation_exception",
> "reason" : "Validation Failed: 1: template is missing;"
> }
> ],
> "type" : "action_request_validation_exception",
> "reason" : "Validation Failed: 1: template is missing;"
> },
> "status" : 400
> }
> 
> ```

I really got confused with this.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 17, 2017, 10:33am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/14 "2017-11-17T10:33:17Z")

</div>

Which version of ES?

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 17, 2017, 10:36am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/15 "2017-11-17T10:36:18Z")

</div>

The ES version is 5.6.4  
"version" : {  
"number" : "5.6.4",  
"build\_hash" : "--- ",  
"build\_date" : "---",  
"build\_snapshot" : false,  
"lucene\_version" : "6.6.1"  
},

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 17, 2017, 10:37am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/16 "2017-11-17T10:37:07Z")

</div>

Make sure you follow the index template documentation for your version of ES: [https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-templates.html)

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 17, 2017, 10:45am UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/17 "2017-11-17T10:45:45Z")

</div>

Ok now it works, the template is created

```
PUT _template/tfjuni
{
  "template": "tfjuni",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "location": {
          "type": "geo_point"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z YYYY"
        }
      }
    }
  }
}

```

Again the field location doesn't change to geo\_point type.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 17, 2017, 12:05pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/18 "2017-11-17T12:05:53Z")

</div>

What do the mappings of a newly created index look like?

---

<div class="post-metadata">

**Author:** ![ahmedcharef](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\
**Post date:** [November 17, 2017, 1:04pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/19 "2017-11-17T13:04:18Z")

</div>

I have a field with lan/lat coordinates.  
I need to map this field as geo\_point in ES,  
As you said in the other post it is possible by using an index template.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [November 17, 2017, 3:29pm UTC](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615/20 "2017-11-17T15:29:32Z")

</div>

Please answer my question. What do the mappings of a newly created index look like? Use Elasticsearch's get mapping API. Please also show an example document. Copy/paste from the JSON tab in Kibana.

[Next page](https://discuss.elastic.co/t/insert-longitude-and-latitude-to-logstash-config-file/107615.md?page=2)