alexevon
(Alex)
January 14, 2019, 9:21pm
1
Hi all-
I have a fairly simple question which I hope not too many asked about:
I have a sample JSON stream from kafka into logstash that looks like this example data:
{
"_index": "kafka_logstash",
"_type": "doc",
"_id": "bGM6TmgBaUYXOpKs-Wzc",
"_version": 1,
"_score": null,
"_source": {
"type": "kafkaconsumer",
"@timestamp ": "2019-01-14T21:18:30.693Z",
"payload": {
"idinstance_locations": 56275,
"longitude": 110,
"modified_date": 1547482708000,
"heading": null,
"unique_id": "3df33695-1322-47b0-abb9-b9182f05cc92",
"latitude": 32,
"device_id": "TestData 5",
"altitude": 999.999,
"speed": null,
"created_date": 1547482708000
}
},
What I wanted to do is just to extract the longitude and latitude from the above and populate them into the location geo-point field...
Is there an mutate/grok filter that can facilitate this?
Thank you!
Alex
Badger
January 14, 2019, 9:53pm
2
You can use mutate
mutate{
rename=>["latitude","location[lat]"]
rename=>["longitude","location[lon]"]
}
The harder part is making sure your index template establishes a mapping that causes location to be a geo_point. This thread might help you with that.
alexevon
(Alex)
January 15, 2019, 2:44am
3
Thank you @Badger for the reply. I added your suggestions to the log.conf file and restarted logstash, but I did not see any changes. log.conf:
filter {
mutate{
rename=>["latitude","location[lat]"]
rename=>["longitude","location[lon]"]
}
}
I added the geo-point as suggested as well:
{
"mapping": {
"doc": {
"properties": {
"@timestamp ": {
"type": "date"
},
"location": {
"type": "geo_point"
},
Is there another mapping perhaps I am missing?
Thank you again!
Badger
January 15, 2019, 1:16pm
4
Do 'GET _template/yourIndexName'. Do you see a geo_point mapping for location?
alexevon
(Alex)
January 15, 2019, 1:48pm
5
I do not actually: syntax: curl -sX GET http://localhost:9200/_template/kafka_logstash
..and the response:
{}
However, here is my index mapping per elastic:
{
"mapping": {
"doc": {
"properties": {
"@timestamp ": {
"type": "date"
},
"location": {
"type": "geo_point"
},
"payload": {
"properties": {
"altitude": {
"type": "float"
},
"created_date": {
"type": "long"
},
"device_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"idinstance_locations": {
"type": "long"
},
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
},
"modified_date": {
"type": "long"
},
"unique_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
Badger
January 15, 2019, 1:56pm
6
Not sure. You might want to ask about the mapping in the elasticsearch forum.
system
(system)
Closed
February 12, 2019, 2:19pm
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.