We have a db table that represents sessions, and are using the jdbc plugin to import into LS. We also have various logs related to the session (whose lines contain the same session id as in the db table) that we are using the file input plugin for. We are still trying to determine whether nested objects or parent-child is the best way to go, but I am running into roadblocks trying to make this work; namely, that LS doesn't seem to currently support parent-child at all, and that trying to get nested relationships to work with this particular scenario is either very difficult or impossible as well. On top of that, I am discovering that Kibana 4 doesn't seem to currently support visualization of nested or parent-child aggregations (even though purportedly the query syntax itself is supported). Has anyone else dealt with a similar scenario, and if so, how did you approach your solution?
What sort of analysis do you want to do here?
We are trying to provide monitoring of our sessions to the end user; some of the necessary fields are in the db, others are only in the logs, so we have to establish a one-to-many relationship somehow, or else come up with a nesting scheme that makes sense. I am reading that Kibana 4 doesn't handle these sorts of relationships well in visualizations:
I'm still working on trying to get parent-child working in Logstash, but there are issues there too:
I would have thought that combining related data from multiple input sources into a single, cohesive, searchable unit would be commonplace and well provided for in the architecture, but apparently not.
Did you get a solution for this? I'm in the same boat... having a nightmare trying to get relational data into Elasticsearch via Logstash and then being able to make sense of that data with Kibana.
Documentation for this is severely lacking for beginners.