# Help with data structure (small project)
**URL:** https://discuss.elastic.co/t/help-with-data-structure-small-project/230684
**Category:** Elasticsearch
**Created:** [May 1, 2020, 9:10am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684 "2020-05-01T09:10:39Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@freddedotme](https://discuss.elastic.co/u/freddedotme)
#### Post date: [May 1, 2020, 9:10am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/1 "2020-05-01T09:10:39Z")
Hi there!
I'm quite new to ES and would need some guidance from you, if possible.
The scenario is the following:
_I have approximately 20 unique people, all identified by a separate UUID._
_Each person has around 2000 unique statistics coupled with a corresponding value._
An example of raw data would be something like this:
```
id: UUID
name: John Doe
stats: ["jumps" -> 23, "farts" -> 52, "kilometersWalked" -> 33, ...]
```
The statistics would be updated when an certain event triggers it and at a maximum interval of one time per hour. As of now I'm stuck of how to solve this, as in what would be a good way to store this?
_Side note: The statistic types are the same for all people, i.e these 2000 types are pre-defined as well as all values are integers._
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [May 1, 2020, 10:00am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/2 "2020-05-01T10:00:12Z")
I have one question: are the metrics cumulative or each document will report the delta since the last update?
I think the best approach would be to have 2 indices:
- time based indices, which will contain one document per metric update per user
- one entity centric index where you can store the latest event you received from each user
The second one is not strictly necessary, but it can make some operations more handy.
To get the latest stats for a given user, you can query on the user `uuid` ordering by timestamp descending.
Also, Transform Jobs might help to build up stats over time for each user (e.g. average number of steps).
---
### Author:  [@freddedotme](https://discuss.elastic.co/u/freddedotme)
#### Post date: [May 1, 2020, 10:19am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/3 "2020-05-01T10:19:13Z")
Hey, thanks for replying.
My idea was for it to be cumulative. On a certain triggered event an iteration of all statistic of said person will update (if changed). My idea was to do a bulk update on existing fields.
What you think?
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [May 1, 2020, 10:24am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/4 "2020-05-01T10:24:23Z")
You can even overwrite the whole document, indexing via bulk update (if you receive all the updates at once) using the user `uuid` as `_id`.
But you would lose the evolution over time.
---
### Author:  [@freddedotme](https://discuss.elastic.co/u/freddedotme)
#### Post date: [May 1, 2020, 10:27am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/5 "2020-05-01T10:27:07Z")
Oh that sounds neat, is it better to overwrite than update? So I'd create a new document by using the path `/` but what would you recommend to use as data types for my case? I assume choosing `Object` is a must for my case, or is it possible to use something similar to tuples (for all stats on said person)?
The evolution over time is not necessary in my case because what I aim for is to reflect current state only.
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [May 1, 2020, 11:06am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/6 "2020-05-01T11:06:24Z")
I think the best approach is to define a mapping with the correct data types.
Do not use `object`. See more on [our documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html).
E.g.
```auto
PUT usermetrics?include_type_name=true
{
"mappings": {
"_doc": {
"properties": {
"uuid": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"@timestamp": {
"type": "date"
},
"stats": {
"properties": {
"jumps": {
"type": "long"
},
"farts": {
"type": "long"
},
"kilometersWalked": {
"type": "scaled_float",
"scaling_factor": 1000
}
}
}
}
}
}
}
```
> [@freddedotme](#):
>
> Oh that sounds neat, is it better to overwrite than update?
Overwriting on the same `_id` would avoid to Elasticsearch to read what already exists.
---
### Author:  [@freddedotme](https://discuss.elastic.co/u/freddedotme)
#### Post date: [May 1, 2020, 11:09am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/7 "2020-05-01T11:09:14Z")
I see, thanks a lot. I assume my following question is very subjective depending on hardware but is it feasible to have 2000 entries in my `stats` array. I've understood that ES uses a flatten data structure, but is there a good rule of thumb or similar when time complexity goes through the roof?
---
### Author:  [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)
#### Post date: [May 1, 2020, 11:21am UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/8 "2020-05-01T11:21:31Z")
I do not see 2000 entries as a problematic number of fields, especially because they're numeric.
You will have to increment the default number of metrics using [`index.mapping.total_fields.limit`](https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping.html#mapping-limit-settings) and increase it. It is dynamic.
As soon as you want to introduce a new metric, before sending the data with the new metric, update the `mapping` to specify the new field and its type. Then send the new data.
I would not try to normalize data. Especially because you will have one document per user.
To optimize, you could even switch off the storage of the `_source`, but it has some [consequences](https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-source-field.html#disable-source-field).
As all your keys are uniques, I do not see any interest on sending a `list` of `key`/`values`.
Send directly an `hashmap` / `dict`.
E.g.
```auto
{
"uuid": "123", "name": "John", "@timestamp": "...",
"stats" {
"jumps": 1,
"farts":1000,
"kilometersWalked": 1.234
}
}
```
At the end, Elasticsearch will in any way index them as detailed in the mapping because it flattens internally the arrays (except if you use `nested` datatype, but it is not useful on your case).
---
### Author:  [@freddedotme](https://discuss.elastic.co/u/freddedotme)
#### Post date: [May 1, 2020, 12:04pm UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/9 "2020-05-01T12:04:19Z")
Thanks a lot!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 29, 2020, 12:13pm UTC](https://discuss.elastic.co/t/help-with-data-structure-small-project/230684/10 "2020-05-29T12:13:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.