Logstash grok/mutate question

[alexevon (Alex)]

Hi all-

I have a fairly simple question which I hope not too many asked about:

I have a sample JSON stream from kafka into logstash that looks like this example data:

{
"_index": "kafka_logstash",
"_type": "doc",
"_id": "bGM6TmgBaUYXOpKs-Wzc",
"_version": 1,
"_score": null,
"_source": {
"type": "kafkaconsumer",
"@timestamp": "2019-01-14T21:18:30.693Z",
"payload": {
"idinstance_locations": 56275,
"longitude": 110,
"modified_date": 1547482708000,
"heading": null,
"unique_id": "3df33695-1322-47b0-abb9-b9182f05cc92",
"latitude": 32,
"device_id": "TestData 5",
"altitude": 999.999,
"speed": null,
"created_date": 1547482708000
}
},

What I wanted to do is just to extract the longitude and latitude from the above and populate them into the location geo-point field...

Is there an mutate/grok filter that can facilitate this?

Thank you!
Alex

[Badger]

You can use mutate

 mutate{
     rename=>["latitude","location[lat]"]
     rename=>["longitude","location[lon]"]
 }

The harder part is making sure your index template establishes a mapping that causes location to be a geo_point. This thread might help you with that.

[alexevon (Alex)]

Thank you @Badger for the reply. I added your suggestions to the log.conf file and restarted logstash, but I did not see any changes. log.conf:

filter {
mutate{
rename=>["latitude","location[lat]"]
rename=>["longitude","location[lon]"]
}
}

I added the geo-point as suggested as well:

{
"mapping": {
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"location": {
"type": "geo_point"
},

Is there another mapping perhaps I am missing?

Thank you again!

[Badger]

Do 'GET _template/yourIndexName'. Do you see a geo_point mapping for location?

[alexevon (Alex)]

I do not actually: syntax: curl -sX GET http://localhost:9200/_template/kafka_logstash

..and the response:

{}

However, here is my index mapping per elastic:

{
"mapping": {
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"location": {
"type": "geo_point"
},
"payload": {
"properties": {
"altitude": {
"type": "float"
},
"created_date": {
"type": "long"
},
"device_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"idinstance_locations": {
"type": "long"
},
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
},
"modified_date": {
"type": "long"
},
"unique_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

[Badger]

Not sure. You might want to ask about the mapping in the elasticsearch forum.

[alexevon (Alex)]

Thank you @Badger

Alex