How to convert lat/lon data coming from MongoDB to geo_point type?

Elastic Stack Logstash
1

I am trying to index a collection to Elasticsearch from MongoDB.

In MongoDB a document contains a Position field which has in turn lat and lon (float) fields that describe the location of any vehicle.
I want these points to be converted to geo_point so that later on we get better and faster aggregation stuff. I am creating a new field named location via my input config file.

Problem is, new field location is created but it is not of geo_point type.
I have read numerous forums including stackoverflow and there people are suggesting to first create mapping for geo point before the index creation on Elasticsearch via PUT API.

But I want to know if there is any other way via .conf input file, to create geo_point without creating their mapping first in Elasticsearch. Is it possible?

Here is my sample document in MongoDB:

{
    "_id" : ObjectId("59a41f63dba6488b3326834f"),
    "MediaID" : "DVR_Backseat_1_362019210",
    "IsRestrictedView" : true,
    "Position" : {
        "lon" : 67.0486,
        "lat" : 24.8589
     }
}

Here is my .conf input file:

input {
	mongodb {
		uri => 'mongodb://localhost/TestDB'
		placeholder_db_dir => 'E:/logstash-mongodb/'
		placeholder_db_name => 'logstash_sqlite.db'
		collection => 'MediaSummary'
		batch_size => 5000
		generateId => false
		parse_method => "simple"
    }
}

filter {
	mutate {
        rename => { "_id" => "mongo_id" }
	}
	grok {
		match => { "Position" => "^?\"lon\"=>%{NUMBER:lon},\s\"lat\"=>%{NUMBER:lat}" }
	}
	date {
		match => [ "timestamp", "YYYY-MM-dd HH:mm:ss ZZZ" ]
    }
	mutate {
		convert => {"lat" => "float"}
		convert => {"lon" => "float"}
	}
	mutate {
		add_field => { "location" => "%{[lat]}, %{[lon]}" }
	}
	mutate {
		convert => {"IsRestrictedView" => "boolean"}
	}
}

output {
	stdout {
		codec => rubydebug
	}
	elasticsearch {
		hosts => ["127.0.0.1:9200"]
		index => "mediasummary"
		doc_as_upsert => true
		document_id => "%{mongo_id}"
	}
}

Please note that Position field shows up as following in stdout:

"Position" => "{\"lon\"=>67.0487, \"lat\"=>24.859}"

Also note that I have used parse_method => "simple". This is because otherwise boolean variables are not converted correctly.

So what should I do to get location field as geo_point type?

2

If your field is called geoip then the default mapping will make that a geo_point. For any other field name you need a mapping. This thread might help.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.