Mapping by date index

Elastic Stack Elasticsearch
1

Hello everyone,

I'm currently trying to implement a KPI application using NodeJS as a Back-End. You can check my previous topic about it here : Develop a KPI interface with ElasticSearch and ..?.

It's my very first time with ElasticSearch and especially with NodeJS so I followed the tutorial of Ranaan Weber : https://blog.raananweber.com/2015/11/24/simple-autocomplete-with-elasticsearch-and-node-js/.

I've made my own model for my application so I've began to set the content of the initMapping() function but I encountered a silly problem. The model is based on the following structure (I deleted useless data) :

{
    "stats": {
        "details": {
            "2016": {
                "months": {
                    "01": {
                        "weeks": {
                            "01": {
                                "days": {
                                    "01": {
                                        "foo": 258
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

and I don't know how to init the mapping by knowing that :

  • details contains a list of objects indexed by fullyear like details { "2016": {...}, "2015": {...} }
  • months contains a list of objects indexed by month index like months { "01": {...}, "02": {...}, ... }
  • weeks contains a list of objects indexed by week index like weeks { "01": {...}, "02": {...}, ... }
  • days contains a list of objects indexed by day index like days { "01": {...}, "02": {...}, ... }

At the beginning, I wanted to use the nested type but I don't want to have to repeat the model for each year ... I surely failed reading the documentation but at least, I have to admit that I tried before bothering you.

I don't know if I'm clear. don't hesitate if I wasn't enough.

Your help would be cool.
A lost intern.

2

Hi @gigouni,

I edited the json example in your post so its easier to read for others. Regarding your question, I'm not sure I get why you need that deep level of nesting to essentially represent a date and value. Is this structure imposed on you by your application? On the elastic search side I would try to flatten the data as much as possible, for example if you index time based events with a value foo I'd try to model it as

{
          "event_time" : "2016-01-01",
          "foo" : 258
}

and the mapping for this kind of data would be Date and an Integer. But maybe I'm missunderstanding your problem, in that case can you try to rephrase your question?

3

Thanks @cbuescher for your answer and for the json. :wink:

That's right that my application needs some explanations. It's a KPI application which works in two steps. The first one is like a simple dashboard to display some data like the revenue since the creation of the company or even how much orders has been made. Something classical so.

The second is the reason of my "complicated" json. The user have to be able to filter from a beginning date to an ending one. In this interval, I need to calculate data. The filter can be on a single day, and that's why I need to have each day. But if the filter has an interval running on two weeks, to avoid too much calculs, I use "global" vars like you could see here :

"details": {
      "2015": {
        "annual_revenue": 102654371,
        "annual_orders_nb": 36585312,
        "months": {
          "01": {
            "monthly_revenue": 5698790,
            "monthly_orders_nb": 36585312,
            "weeks": {
              "01": {
                "weekly_revenue": 1400000,
                "weekly_orders_nb": 36585312,
                "days": {
                  "01": {
                    "daily_revenue": 1400000,
                    "daily_orders_nb": 365
                  }
                }
              }
            }
          }
        }
      }
    }

At the beginning of the project, I was thinking about using your way, with dates but it wouldn't be performant enough due to all the calculs that need to be done.

Is it clearer ?

(Hope my english isn't too rough too.)

4

ElasticSearch does he allow so no manage this type of architecture?

I would like to have your impression or at least, your verdict about my scheme.

Have a good day.

5

Up :slight_smile:

6

For those who could be interested by a similar case later.
I've handled my problem with the following structure :

"details": [
      {
        "annual_index": 2016,
        "annual_opened_sites": 1564,
        "annual_orders_nb": 36585312,
        "months": [
          {
            "monthly_index": 1,
            "monthly_revenue": 5698790,
            "monthly_orders_nb": 36585312,
            "weeks": [
              {
                "weekly_index": 1,
                "weekly_revenue": 1400000,
                "weekly_orders_nb": 36585312,
                "days": [
                  {
                    "daily_index": 1,
                    "daily_revenue": 1400000,
                    "daily_orders_nb": 365
                  }
                ]
              }
            ]
          }
        ]
      }
    ]

This solution is much simpler. The mapping is now about nested objects. However, I know, calculations will be more greedy because I should be doing loops in my objects.

But thank you for your help @cbuescher. :wink: