Hello,
As a new ES user, I'm facing some questions about data structure, denormalization and what comes with it :
Considering a simple 1:n data structure, like Device
-> Record
, where a Device
produces a high number of Record
(1M).
Trying to denormalize the data (and considering the removal of "types"), I imagined the following 2 indexes mapping on ES :
Index Device
mapping :
{
"mapping": {
"_doc": {
"properties": {
"deviceId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"location": {
"type": "geo_point"
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"upStatus": {
"type": "boolean"
}
}
}
}
}
Index Record
mapping :
{
"mapping": {
"record": {
"properties": {
"timestamp": {
"type": "date"
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"device": {
"properties": {
"deviceId": {
"type": "text"
},
"location": {
"type": "geo_point"
},
"name": {
"type": "text"
},
"upStatus": {
"type": "boolean"
}
}
}
}
}
}
}
I'm facing two problems with this :
- A client that wants to index records have to send all the
Device
properties along with theRecord
-> I expected there was a way to solve this using the ingest node but I could not find a Processor that can fetch the necessaryDevice
properties to add them to theRecord
. - Data integrity : each time a property on
Device
changes, all correspondingRecords
(> 1M) needs to be updated.
Do you have any pointers on those 2 issues or maybe a better data structure, etc… ?
Thank you very much for your inputs !