Unable to save documents by timestamp - only one document is saved and updated everytime


(Nathan Ware) #1

I want to save documents in an Index by timestamp.
I am saving documents every second, so I want to see a document being created every second.
But when I save the document by passing new timestamp value, a new document is not created but the same existing document is updated.
So instead of seeing documents being saved per second, I see the same document getting update every second.
Below is the mapping information about my index.

{
"iotdevice": {
"aliases": {},
"mappings": {
"sensordata": {
"properties": {
"nodeID": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"sensorData": {
"type": "float"
},
"sensorID": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"sensorName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"sensorState": {
"type": "long"
},
"sensorType": {
"type": "long"
},
"siteIP": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}
}
}
},
"settings": {
"index": {
"creation_date": "1508352509436",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "ZxcszX-LTzatqYCQV3z4QA",
"version": {
"created": "5030299"
},
"provided_name": "iotdevice"
}
}
}
}

I am saving the document every second by using the bulk insert operation.

But after saving 100's of documents per second, I am seeing only 1 record with the most recent timestamp/

https://search-sensordata-hima7jdvzx6rkn6tpprlqlyp4e.us-east-1.es.amazonaws.com/iotdevice/_search/?size=1000&pretty=1
Response ->
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "iotdevice",
"_type": "sensordata",
"_id": "ID002",
"_score": 1,
"_source": {
"sensorID": "ID002",
"sensorName": "Sensor_SFO",
"sensorState": 1,
"timestamp": "2017-10-19 16:16:44"
}
}
}
]
}
}


#2

You appear to have set _id (the document id) to be the same as sensorID. That means you will only ever have one document per sensor per index. If you are writing one document per second per sensor then do not set the document id, just let es generate it.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.