How to model ES data to ensure Kibana compatibility?

Elastic Stack Kibana
1

I am new to ES, coming from a RDBMS background and understand that there are certain ways to model your data with ES, that ensure compatibility with Kibana. Namely, nested objects.

I will be using ES for a real-time web application, where the we currently have the main queryable data in a SQL Server data mart. Things like user profiles, etc. are kept in separate, normalized tables.

I would like to know some suggestions on how to model this data in ES. I am guessing that things like user profiles can be kept in separate types?

Taking a stab at this, I envision something like the following mode contained in one index:

 User (type)
      Key (UserID), Name, etc...          
      Geographies
         Key (UserID:GeoID), GeoName
      Projects
         Key (UserID:ProjectID), ProjectName
         ProjectContent
            Key (UserID:ProjectID:ProjectContentID), GeoName, Title, Description, Timestamp 
  ProfileSettings (type)
      Key (UserID:ProfileSettingsID) - Setting1, Setting2, etc...

The ProjectContent type represents the data mart.

Also, simply guessing on how to create unique keys, I would also like some advice on key naming conventions.

Thank you.

2

I'd recommend separating different types of data into separate indexes, if possible. Types have some namespace limitations you should be aware of: for instance, if two types in the same index define a field with the same name in their mappings, that field needs to have the same schema characteristics (e.g. if"id" in type1 is an integer, it must also be an integer in type2). This is not a hard-and-fast rule in ES 1.x, but it will be starting with 2.x. More info on that here: https://www.elastic.co/blog/great-mapping-refactoring

In terms of nested documents, if you plan to use Kibana to visualize that data, I'd recommend flattening the namespace. Kibana currently does not support aggregations on top of nested documents (unless you use an "include_in_parent" setting, which does that sort of flattening for you at index time): https://github.com/elastic/kibana/issues/1084