Hi all,
I'm trying to ingest into an index my data (json) with a HTTP POST. Everything works fine but I have a geo point like this { "point_location" : "11.0500,-0.2500" } which is not recognized as a geo point but like a simple text. I have another field which is a number and this one is recognized like a text too. and not as a number.
How can I fix it ?
Thx.
Hi @Phildefer
You need to create mappings ahead of time to make sure you get the types you expect.
You will need to show us your mapping and exact sample documents so that we can see / test
Thx @stephenb for your answer.
I use N8N to manage my data (Get the json via an api and a http request) a node to create the index with this mapping :
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"actor1": {
"type": "text"
},
"actor2": {
"type": "text"
},
"admin1": {
"type": "keyword"
},
"admin2": {
"type": "keyword"
},
"admin3": {
"type": "keyword"
},
"assoc_actor_1": {
"type": "text"
},
"assoc_actor_2": {
"type": "text"
},
"civilian_targeting": {
"type": "keyword"
},
"country": {
"type": "keyword"
},
"disorder_type": {
"type": "keyword"
},
"event_date": {
"type": "keyword"
},
"event_id_cnty": {
"type": "keyword"
},
"event_type": {
"type": "keyword"
},
"fatalities": {
"type": "long"
},
"geo_precision": {
"type": "long"
},
"inter1": {
"type": "keyword"
},
"inter2": {
"type": "keyword"
},
"interaction": {
"type": "keyword"
},
"iso": {
"type": "long"
},
"latitude": {
"type": "double"
},
"location": {
"type": "keyword"
},
"longitude": {
"type": "double"
},
"notes": {
"type": "text"
},
"point_location": {
"type": "geo_point"
},
"region": {
"type": "keyword"
},
"source": {
"type": "text"
},
"source_scale": {
"type": "keyword"
},
"sub_event_type": {
"type": "keyword"
},
"tags": {
"type": "text"
},
"time_precision": {
"type": "long"
},
"timestamp": {
"type": "date",
"format": "epoch_second"
},
"year": {
"type": "long"
}
}
}
}
and a HTTP request Node to send the json to elasticsearch.
What does the incoming JSON look like and which fields are not correct?
Mapping is just part of the puzzle
here is an exemple of the json :
{
"event_id_cnty":
"GHA2450",
"event_date":
"2024-11-29",
"year":
"2024",
"time_precision":
"1",
"disorder_type":
"Political violence",
"event_type":
"Battles",
"sub_event_type":
"Armed clash",
"actor1":
"Kusasis Ethnic Militia (Ghana)",
"assoc_actor_1":
"",
"inter1":
"Identity militia",
"actor2":
"Mamprusi Ethnic Militia (Ghana)",
"assoc_actor_2":
"",
"inter2":
"Identity militia",
"interaction":
"Identity militia-Identity militia",
"civilian_targeting":
"",
"iso":
"288",
"region":
"Western Africa",
"country":
"Ghana",
"admin1":
"Upper East",
"admin2":
"Bawku",
"admin3":
"",
"location":
"Bawku",
"latitude":
"11.0500",
"longitude":
"-0.2500",
"geo_precision":
"1",
"source":
"Citi News",
"source_scale":
"National",
"notes":
"On 29 November 2024, Kusasis and Mamprusi ethnic militia clashed, in the continued chieftaincy dispute, around the Gingande-Zongo area in Bawku city (Bawku, Upper East). Three people were killed in the clash, an unspecified number of people were seriously injured, and houses were set ablaze.",
"fatalities":
"3",
"tags":
"",
"timestamp":
"1733203849",
"point_location":
"11.0500,-0.2500"
},
The fields Fatalities are in text format and not into a number format in elasticsearch and the point_location is not recognized as a geo_point.
Looks good to me I suspect you are confusing _source and fields
_source how the data comes in
fields A representation of how the data is stored.
Note in the fields section
"point_location": [
{
"coordinates": [
-0.25,
11.05
],
"type": "Point"
}
],
Here we go...
PUT discuss-geo
{
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"actor1": {
"type": "text"
},
"actor2": {
"type": "text"
},
"admin1": {
"type": "keyword"
},
"admin2": {
"type": "keyword"
},
"admin3": {
"type": "keyword"
},
"assoc_actor_1": {
"type": "text"
},
"assoc_actor_2": {
"type": "text"
},
"civilian_targeting": {
"type": "keyword"
},
"country": {
"type": "keyword"
},
"disorder_type": {
"type": "keyword"
},
"event_date": {
"type": "keyword"
},
"event_id_cnty": {
"type": "keyword"
},
"event_type": {
"type": "keyword"
},
"fatalities": {
"type": "long"
},
"geo_precision": {
"type": "long"
},
"inter1": {
"type": "keyword"
},
"inter2": {
"type": "keyword"
},
"interaction": {
"type": "keyword"
},
"iso": {
"type": "long"
},
"latitude": {
"type": "double"
},
"location": {
"type": "keyword"
},
"longitude": {
"type": "double"
},
"notes": {
"type": "text"
},
"point_location": {
"type": "geo_point"
},
"region": {
"type": "keyword"
},
"source": {
"type": "text"
},
"source_scale": {
"type": "keyword"
},
"sub_event_type": {
"type": "keyword"
},
"tags": {
"type": "text"
},
"time_precision": {
"type": "long"
},
"timestamp": {
"type": "date",
"format": "epoch_second"
},
"year": {
"type": "long"
}
}
}
}
POST discuss-geo/_doc
{
"event_id_cnty": "GHA2450",
"event_date": "2024-11-29",
"year": "2024",
"time_precision": "1",
"disorder_type": "Political violence",
"event_type": "Battles",
"sub_event_type": "Armed clash",
"actor1": "Kusasis Ethnic Militia (Ghana)",
"assoc_actor_1": "",
"inter1": "Identity militia",
"actor2": "Mamprusi Ethnic Militia (Ghana)",
"assoc_actor_2": "",
"inter2": "Identity militia",
"interaction": "Identity militia-Identity militia",
"civilian_targeting": "",
"iso": "288",
"region": "Western Africa",
"country": "Ghana",
"admin1": "Upper East",
"admin2": "Bawku",
"admin3": "",
"location": "Bawku",
"latitude": "11.0500",
"longitude": "-0.2500",
"geo_precision": "1",
"source": "Citi News",
"source_scale": "National",
"notes": "On 29 November 2024, Kusasis and Mamprusi ethnic militia clashed, in the continued chieftaincy dispute, around the Gingande-Zongo area in Bawku city (Bawku, Upper East). Three people were killed in the clash, an unspecified number of people were seriously injured, and houses were set ablaze.",
"fatalities": "3",
"tags": "",
"timestamp": "1733203849",
"point_location": "11.0500,-0.2500"
}
GET discuss-geo/_search
{
"fields": [
"*"
]
}
Result
{
"took": 36,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1,
"hits": [
{
"_index": "discuss-geo",
"_id": "fXq2npMBS9VyZaU18gvx",
"_score": 1,
"_source": {
"event_id_cnty": "GHA2450",
"event_date": "2024-11-29",
"year": "2024",
"time_precision": "1",
"disorder_type": "Political violence",
"event_type": "Battles",
"sub_event_type": "Armed clash",
"actor1": "Kusasis Ethnic Militia (Ghana)",
"assoc_actor_1": "",
"inter1": "Identity militia",
"actor2": "Mamprusi Ethnic Militia (Ghana)",
"assoc_actor_2": "",
"inter2": "Identity militia",
"interaction": "Identity militia-Identity militia",
"civilian_targeting": "",
"iso": "288",
"region": "Western Africa",
"country": "Ghana",
"admin1": "Upper East",
"admin2": "Bawku",
"admin3": "",
"location": "Bawku",
"latitude": "11.0500",
"longitude": "-0.2500",
"geo_precision": "1",
"source": "Citi News",
"source_scale": "National",
"notes": "On 29 November 2024, Kusasis and Mamprusi ethnic militia clashed, in the continued chieftaincy dispute, around the Gingande-Zongo area in Bawku city (Bawku, Upper East). Three people were killed in the clash, an unspecified number of people were seriously injured, and houses were set ablaze.",
"fatalities": "3",
"tags": "",
"timestamp": "1733203849",
"point_location": "11.0500,-0.2500"
},
"fields": {
"country": [
"Ghana"
],
"notes": [
"On 29 November 2024, Kusasis and Mamprusi ethnic militia clashed, in the continued chieftaincy dispute, around the Gingande-Zongo area in Bawku city (Bawku, Upper East). Three people were killed in the clash, an unspecified number of people were seriously injured, and houses were set ablaze."
],
"disorder_type": [
"Political violence"
],
"event_id_cnty": [
"GHA2450"
],
"iso": [
288
],
"year": [
2024
],
"latitude": [
11.05
],
"source": [
"Citi News"
],
"sub_event_type": [
"Armed clash"
],
"event_type": [
"Battles"
],
"event_date": [
"2024-11-29"
],
"time_precision": [
1
],
"longitude": [
-0.25
],
"timestamp": [
"1733203849"
],
"geo_precision": [
1
],
"inter1": [
"Identity militia"
],
"inter2": [
"Identity militia"
],
"source_scale": [
"National"
],
"assoc_actor_1": [
""
],
"civilian_targeting": [
""
],
"assoc_actor_2": [
""
],
"tags": [
""
],
"point_location": [
{
"coordinates": [
-0.25,
11.05
],
"type": "Point"
}
],
"actor2": [
"Mamprusi Ethnic Militia (Ghana)"
],
"actor1": [
"Kusasis Ethnic Militia (Ghana)"
],
"fatalities": [
3
],
"admin1": [
"Upper East"
],
"interaction": [
"Identity militia-Identity militia"
],
"admin2": [
"Bawku"
],
"location": [
"Bawku"
],
"region": [
"Western Africa"
],
"admin3": [
""
]
}
}
]
}
}
Thx so much it works!
oups no there is a problem i just noticed that I have a problem with the dates. I need to desactivate the date filter to see the points on the map. It seems that the dates are good (event_date) but i can see that @timestamp is empty.
You don't have a
@timestamp field
You have a
timestamp
Field in your source document, perhaps I am missing.
So I can just delete the @timestamp in the mapping ?
I thought this field was mandatory to manage the time in the dashboard or in the maps.
Sure.
What do you want to do?
You will need to create a Data View and set the time field to your timestamp if you want to use that in the map .. and filter by time
I didn't know that it was possible to change the time field in the dataview that's why I thought @timestamp was mandatory. So it's ok now. Thx so much!
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.