Hi,
I'm looking at using elasticsearch for a project I'm working on. I think it'll be a good fit because I want to use elasticsearch as a place to aggregate lots of different data sources to help centralise my data. I'm using RobotFramework and storing the output in a MongoDB. It's been a bit painful, but I've managed to start syncing across the results into my elasticsearch via logstash. I'm trying to view these results in Kibana, but I'm only seeing 55 results (out of 370) and these are missing the testing data (the data is sat alongside other data that helps classify it). I suspect this is because I haven't providing a mapping for the data which has a bizarre and recursive structure and that this has upset Kibana and caused it to ignore some records.
I've only just started using ElasticSearch and I'm quite keen to stick with it. The robot testing data has a JSON schema: https://github.com/Lemonlemmings/robotframework/blob/master/doc/schema/robot-10.schema.json and I have a gist which shows an example of what (a fairly simple) set of results actually looks like as an object: https://gist.github.com/Lemonlemmings/40f137050f00d72613a84504091fd0ef
The gist only shows the test results in my JSON document, and is missing the rest of the structure which is static (since this is the problematic part I figure it's worth pursuing). I have read around and see nested structures (which looks like they could help with the issue) but ultimately, given the large size of these structures would probably be quite unwieldly (and inefficient). Would anyone have any suggestions as to why I'm suffering these issues in Kibana? And secondly does anyone have any suggestions for a sensible way to do the mapping?
I'd like to be able to search through the base level of these results ideally (the first layer to see what test cases have been run). I could just trim it so I had only this first layer, but if there's a better solution than I'd rather go for that. I don't know if there's a way to map the first layer and then tell do a mapping that asks elasticsearch to not bother indexing the rest?