I'm new to Elasticsearch but have done enough research to figure out how to map, enter and perform basic "relational" searches using Nested and Parent/Child structures. I believe my data is better suited to Parent/Child structures but I also desire support for aggregation in Kibana. My understanding is that in v4.4 this will be supported for Nested structures and the files are available now (https://github.com/elastic/kibana/pull/4806).
I have a lot of detailed questions but for simplicity I think it might be better if I simply give an example of my data structure and ask for the approach you would take to getting what I desire from Elasticsearch (ES) and Kibana.
My data is contained in large CSV files (~1MB/file to 30+GB/file), related to each other by an ID variable. A very simplified example of the structure for 3 CSV files is:
ID,AGE,GENDER,WAVE 1,49,M,1 2,72,F,0
ID,TIME,EVENT1 1,4/20/2095,V39 1,4/21/2095,T21 2,5/17/2094,V39 2,5/18/2094,R9 2,5/20/2094,Q20
ID,TIME,EVENT2 1,4/22/2095,P90 1,4/23/2095,T90 1,4/23/2095,E2 1,4/24/2095,RR2 2,5/18/2094,E2 2,5/18/2094,W3
A simple Elasticsearch I'd like to be able to perform is: return any ID who has WAVE=1 and EVENT1=V39 and EVENT2=E2. I don't think this will work for a purely flat structure since I wouldn't be able to express that WAVE=1, EVENT1=V39 and EVENT2=E2 must be true for the same ID. Therefore I need to use either Nested or Parent/Child structures.
A simple Kibana plot could be a pie chart of GENDER, aggregated by EVENT1. I know that flattening out my structure (so all variables (EVENT1, EVENT2, etc) are in the same CSV) would allow for this but if you imagine the above stucture but with 10,000+ ID's, each with 100's of rows of events (plus many more EVENTX variables) you realize it would be difficult to manage.
Given this example and what I'd like to get from the data do you recommend using Nested or Parent/Child structures? If I used Parent/Child structures could I get aggregated plots for variables across the CSV's (i.e. across the parents/children)? If I use Nested structures is there a good way to import the data or will I need to flatten my CSV's first and then assign the nested variables?