Nested Document vs Single/Separate Document query!

Elastic Stack Elasticsearch
1

Not sure , whether this question is asked already. But I couldn't find similar question and answer.

Currently I'm preparing the log structure to store it in Elasticsearch. for that, I would like to know which one is perfect for long time search [ 1 year data ] "Nested document" or "Single Document".
Whether I could see any slowness when query nested document over 1 year time period.

Which one I would prefer for my log structure either Nested or Single document ?

Below is the example of SINGLE Nested Document:

{"web1":{"test01":{"Uptime":277887,"Name":"test01","TotalOutput":0,"TotalOutputRate":0,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":0},"TotalConnections":0,"TotalInputRate":0,"TotalInput":0},"Uptime":606934,"TotalInputRate":611695,"app102_02":{"Uptime":500891,"Name":"app102_02","TotalOutput":0,"TotalOutputRate":0,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":0},"TotalConnections":0,"TotalInputRate":0,"TotalInput":0},"timestamp":"2016-06-20 09:53:34","app102_01":{"Uptime":336537,"Name":"app102_01","TotalOutput":254245747,"TotalOutputRate":92544,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":1},"TotalConnections":1,"TotalInputRate":91536,"TotalInput":16542639433},"test02":{"Uptime":18346,"Name":"test02","TotalOutput":0,"TotalOutputRate":0,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":0},"TotalConnections":0,"TotalInputRate":0,"TotalInput":0},"app203_01":{"Uptime":65969,"Name":"app203_01","TotalOutput":5622464792,"TotalOutputRate":85467,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":1},"TotalConnections":1,"TotalInputRate":84711,"TotalInput":5567492307},"app202_01":{"Uptime":6050,"Name":"app202_01","TotalOutput":298164476,"TotalOutputRate":88098,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":1},"TotalConnections":1,"TotalInputRate":87159,"TotalInput":528620329},"app31":{"Uptime":18424,"Name":"app31","TotalOutput":0,"TotalOutputRate":0,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":0},"TotalConnections":0,"TotalInputRate":0,"TotalInput":0},"app101_01":{"Uptime":5897,"Name":"app101_01","TotalOutput":488257431,"TotalOutputRate":83028,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":1},"TotalConnections":1,"TotalInputRate":82129,"TotalInput":484619739},"TotalInput":355511505764,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"DVRCHUNKS":0,"R1":3,"WEBM":0,"R2":7},"TotalOutput":563728749361,"cambtest01_01":{"Uptime":181258,"Name":"cambtest01_01","TotalOutput":6373160103,"TotalOutputRate":183271,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":1,"R2":1},"TotalConnections":2,"TotalInputRate":91219,"TotalInput":16533919303},"TotalOutputRate":721560,"App":"web1","TotalConnections":10,"app204_01":{"Uptime":65970,"Name":"app204_01","TotalOutput":5622069756,"TotalOutputRate":85059,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":1},"TotalConnections":1,"TotalInputRate":84218,"TotalInput":5567236176},"app201_01":{"Uptime":65980,"Name":"app201_01","TotalOutput":5823095542,"TotalOutputRate":87995,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":0,"R2":1},"TotalConnections":1,"TotalInputRate":87503,"TotalInput":5766095725},"test02wheel":{"Uptime":18357,"Name":"test02wheel","TotalOutput":0,"TotalOutputRate":0,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":1,"R2":0},"TotalConnections":1,"TotalInputRate":0,"TotalInput":0},"app32":{"Uptime":18380,"Name":"app32","TotalOutput":0,"TotalOutputRate":0,"Cal":{"C1":0,"S1":0,"M1":0,"S1":0,"R1":1,"R2":0},"TotalConnections":1,"TotalInputRate":0,"TotalInput":0}},"@version":"1","@timestamp":"2016-06-20T09:53:36.075Z","host":"10.11.12.202","port":38229,"type":"application1" }

2

Please format your code.

It depends on what you are searching for. As a response, are you expecting to see the full document or only a single line of the document?

If the later, I'd copy into each single line, the "parent" values.

If the former, I'd go for nested docs.

3

Sorry I tried to upload the code in formatted one. But it not accepted more than 5000 lines in forum. that's why I updated as a minify json.

It depends on what you are searching for. As a response, are you expecting to see the full document or only a single line of the document?

  • I expected a response as a single value from the document like value 277887 from "web1.test01.uptime".

If the later, I'd copy into each single line, the "parent" values.
If the former, I'd go for nested docs.

  •  Sorry,I couldn't understand. Please share more info.
4

I'd index data like:

PUT index/doc/test01
{
   "Uptime":277887,
   "Name":"test01",
   "TotalOutput":0,
   "TotalOutputRate":0,
   "Cal":{
      "C1":0,
      "S1":0,
      "M1":0,
      "S1":0,
      "R1":0,
      "R2":0
   },
   "TotalConnections":0,
   "TotalInputRate":0,
   "TotalInput":0
}

Warning: S1 is a duplicated key, which is wrong.

5

Sorry, that should be S1 and S2. May be I posted a testing data that still not finalized to insert into Elasticsearch.

Could you please share which one will be fit for long term [ 1 year ] ?

6

I wrote this: Nested Document vs Single/Separate Document query! - #4 by dadoonet

7

Do you mean, keep it as a separate documents instead of put entire thing in a single document ?.

8

Yes. That's what I meant.