Help with data structure (small project)

[freddedotme]

Hi there!

I'm quite new to ES and would need some guidance from you, if possible.
The scenario is the following:

I have approximately 20 unique people, all identified by a separate UUID.
Each person has around 2000 unique statistics coupled with a corresponding value.

An example of raw data would be something like this:

id: UUID
name: John Doe
stats: ["jumps" -> 23, "farts" -> 52, "kilometersWalked" -> 33, ...]

The statistics would be updated when an certain event triggers it and at a maximum interval of one time per hour. As of now I'm stuck of how to solve this, as in what would be a good way to store this?

Side note: The statistic types are the same for all people, i.e these 2000 types are pre-defined as well as all values are integers.

[Luca_Belluccini]

I have one question: are the metrics cumulative or each document will report the delta since the last update?

I think the best approach would be to have 2 indices:

• time based indices, which will contain one document per metric update per user
• one entity centric index where you can store the latest event you received from each user

The second one is not strictly necessary, but it can make some operations more handy.
To get the latest stats for a given user, you can query on the user uuid ordering by timestamp descending.

Also, Transform Jobs might help to build up stats over time for each user (e.g. average number of steps).

[freddedotme]

Hey, thanks for replying.

My idea was for it to be cumulative. On a certain triggered event an iteration of all statistic of said person will update (if changed). My idea was to do a bulk update on existing fields.

What you think?

[Luca_Belluccini]

You can even overwrite the whole document, indexing via bulk update (if you receive all the updates at once) using the user uuid as _id.
But you would lose the evolution over time.

[freddedotme]

Oh that sounds neat, is it better to overwrite than update? So I'd create a new document by using the path <identifier>/<uuid> but what would you recommend to use as data types for my case? I assume choosing Object is a must for my case, or is it possible to use something similar to tuples (for all stats on said person)?

The evolution over time is not necessary in my case because what I aim for is to reflect current state only.

[Luca_Belluccini]

I think the best approach is to define a mapping with the correct data types.
Do not use object. See more on our documentation.

E.g.

PUT usermetrics?include_type_name=true
{
  "mappings": {
    "_doc": {
      "properties": {
        "uuid": {
          "type": "keyword"
        },
        "name": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date"
        },
        "stats": {
          "properties": {
            "jumps": {
              "type": "long"
            },
            "farts": {
              "type": "long"
            },
            "kilometersWalked": {
              "type": "scaled_float",
              "scaling_factor": 1000
            }
          }
        }
      }
    }
  }
}

@freddedotme

Oh that sounds neat, is it better to overwrite than update?

Overwriting on the same _id would avoid to Elasticsearch to read what already exists.

[freddedotme]

I see, thanks a lot. I assume my following question is very subjective depending on hardware but is it feasible to have 2000 entries in my stats array. I've understood that ES uses a flatten data structure, but is there a good rule of thumb or similar when time complexity goes through the roof?

[Luca_Belluccini]

I do not see 2000 entries as a problematic number of fields, especially because they're numeric.

You will have to increment the default number of metrics using index.mapping.total_fields.limit and increase it. It is dynamic.

As soon as you want to introduce a new metric, before sending the data with the new metric, update the mapping to specify the new field and its type. Then send the new data.

I would not try to normalize data. Especially because you will have one document per user.

To optimize, you could even switch off the storage of the _source, but it has some consequences.

As all your keys are uniques, I do not see any interest on sending a list of key/values.
Send directly an hashmap / dict.

E.g.

{
  "uuid": "123", "name": "John", "@timestamp": "...",
  "stats" {
    "jumps": 1,
    "farts":1000,
    "kilometersWalked": 1.234
  }
}

At the end, Elasticsearch will in any way index them as detailed in the mapping because it flattens internally the arrays (except if you use nested datatype, but it is not useful on your case).

[freddedotme]

Thanks a lot!