# Aggregate vs facets vs nested documents? **URL:** \ **Category:** Elasticsearch\ **Created:** [December 23, 2013, 6:39pm UTC](https://discuss.elastic.co/t/aggregate-vs-facets-vs-nested-documents/15004 "2013-12-23T18:39:44Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Vijay\_J](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijay_j/32/1904_2.png) [@Vijay\_J](https://discuss.elastic.co/u/Vijay_J)\ **Post date:** [December 23, 2013, 6:39pm UTC](https://discuss.elastic.co/t/aggregate-vs-facets-vs-nested-documents/15004/1 "2013-12-23T18:39:44Z")
We just started to use logstash for parsing logs and adding the data to ElasticSearch. Using kibana, we can get some good visualizations on our process runs I have the records in ES as follows ( these are basically log entries from log stash) Timestamp TxnId Caller EventName Duration Other Data.. 2013-12-20T22:35:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7cf45 Worker Start .. 2013-12-20T22:38:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7abcd Worker Start .. 2013-12-20T22:40:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7cf45 Worker End ``` 20 ``` .. 2013-12-20T22:42:17.109365+00:01 9f3c264b-8ee3-4b2a-ac16-162290c7abcd Worker End ``` 40 ``` .. 2013-12-20T22:45:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7efgh Worker Start .. Basically this is a log of worker process and their start and finish timesFor a given time period, we would like to know all worker processes that have started and ended and worker process that have not ended. For e.g.: the result for the data above would be Timestamp TxnId Caller EventName Other Duration Other Data.. 2013-12-20T22:35:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7cf45 Worker Start Ended 20 .. 2013-12-20T22:38:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7abcd Worker Start Ended 40 .. 2013-12-20T22:45:17.109365+00:00 9f3c264b-8ee3-4b2a-ac16-162290c7efgh Worker Start .. This can be achieved in sql by grouping on TxnId. We could ideally filter this further with having to find workers that have not ended ( and how long they have been working/timeout). I think this is basically a co-relation on TxnId and EventName. I have looked at facets (and aggregations) , nested documents, parent child objects. I did not find lot of documentation and examples on new way of faceting ( aggregations). It seems to me that it would be ideal if I could build another index that uses TxnId as co-relation and then I can get the results I want. I guess this is pretty close to what facets can do. I am still not sure that if I facet these documents by TxnId would I be able to run a query across all faceted entries asking for start time and end time. Can I build an alternate index of nested documents based on TxnId ? If so, I can pretty much query what I want . The other alternative is to update the document with appropriate data while reading from log entries. It would be great if someone can point me in the right direction here. Thanks, Vijay -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6c80a812-7c79-4794-98ca-c13e367302b6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c80a812-7c79-4794-98ca-c13e367302b6%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [December 24, 2013, 1:39pm UTC](https://discuss.elastic.co/t/aggregate-vs-facets-vs-nested-documents/15004/2 "2013-12-24T13:39:26Z")
Hi, Your data model is assuming a 1-N relationship between transactions and worker events. There several ways that you can solve this kind of issues with Elasticsearch: - denormalization, - nested documents, - parent/child relations. Maybe the easiest way to do it would be to store data directly in the expected format. On a start event, you would insert a new transaction into your index using the transaction id as a document id, and later when the transaction ends, you could update the transaction to record the fact that the transaction finished and its duration. With this option, data is indexed in a way that is easily searchable and you'll be able to leverage all the power of aggregations facets to compute things like the number of non-terminated transactions, the average duration, etc. However, if you have a very high ingestion rate, this option might become a bit too slow, in which case you might want instead to record transactions as parent documents and events as child documents using parent/child relations. This will make indexing faster but require more memory and you'll have less power at query time. For example, finding transactions that started but didn't finished would require to find the start events, then resolve their transaction ids, then find the end events, resolve their transaction ids and finally to compute the difference of the two sets of transactions. This kind of query would typically execute much slower than if you already had all information in a single transaction document (as with the previous option). Moreover computing things like the average duration of transactions using facets or aggregations wouldn't be possible anymore with parent/child relations. See [http://www.elasticsearch.org/blog/managing-relations-inside-elasticsearch/for](http://www.elasticsearch.org/blog/managing-relations-inside-elasticsearch/for) more information. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7c9-M999xebmfMgrhv%2BpA51R1EeF0oTuZ5hXycf0y0DA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7c9-M999xebmfMgrhv%2BpA51R1EeF0oTuZ5hXycf0y0DA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vijay\_J](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vijay_j/32/1904_2.png) [@Vijay\_J](https://discuss.elastic.co/u/Vijay_J)\ **Post date:** [December 24, 2013, 7:16pm UTC](https://discuss.elastic.co/t/aggregate-vs-facets-vs-nested-documents/15004/3 "2013-12-24T19:16:44Z")
Adrien, Thanks for the response. Since the rate of log generation is not too higher at the moment, I will try option 1 at the moment. Really appreciate all the help. Regards, Vijay PS - Sent from my phone > On Dec 24, 2013, at 5:39 AM, Adrien Grand [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com) wrote: > > Hi, > > Your data model is assuming a 1-N relationship between transactions and worker events. There several ways that you can solve this kind of issues with Elasticsearch: > > - denormalization, > - nested documents, > - parent/child relations. > > Maybe the easiest way to do it would be to store data directly in the expected format. On a start event, you would insert a new transaction into your index using the transaction id as a document id, and later when the transaction ends, you could update the transaction to record the fact that the transaction finished and its duration. With this option, data is indexed in a way that is easily searchable and you'll be able to leverage all the power of aggregations facets to compute things like the number of non-terminated transactions, the average duration, etc. > > However, if you have a very high ingestion rate, this option might become a bit too slow, in which case you might want instead to record transactions as parent documents and events as child documents using parent/child relations. This will make indexing faster but require more memory and you'll have less power at query time. For example, finding transactions that started but didn't finished would require to find the start events, then resolve their transaction ids, then find the end events, resolve their transaction ids and finally to compute the difference of the two sets of transactions. This kind of query would typically execute much slower than if you already had all information in a single transaction document (as with the previous option). Moreover computing things like the average duration of transactions using facets or aggregations wouldn't be possible anymore with parent/child relations. > > See [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/managing-relations-inside-elasticsearch/) for more information. > > ## -- Adrien Grand > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/sAm6gAXIxW4/unsubscribe](https://groups.google.com/d/topic/elasticsearch/sAm6gAXIxW4/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7c9-M999xebmfMgrhv%2BpA51R1EeF0oTuZ5hXycf0y0DA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7c9-M999xebmfMgrhv%2BpA51R1EeF0oTuZ5hXycf0y0DA%40mail.gmail.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4917E984-F2B9-4C02-B39C-7AE5A6E227FD%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/4917E984-F2B9-4C02-B39C-7AE5A6E227FD%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:59am UTC](https://discuss.elastic.co/t/aggregate-vs-facets-vs-nested-documents/15004/4 "2017-07-06T01:59:28Z")