# Terms facet calculation path (random vs sequential disk access) **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2013, 12:31pm UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564 "2013-04-12T12:31:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ahmet\_Akyol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmet_akyol/32/2389_2.png) [@Ahmet\_Akyol](https://discuss.elastic.co/u/Ahmet_Akyol)\ **Post date:** [April 12, 2013, 12:31pm UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/1 "2013-04-12T12:31:08Z")
Hi, When calculating terms facet, does ES have to look actual data( aka source which means random access), other than internal index (which may offer sequential read, I guess). So, my real intention is to understand, how much random vs sequential access to disk occurs during this type facet calculation. Of course, it depends on docs and query. So here is a simple example: a sample doc of many: { "userid":"uid\_1", // as \_id , not\_analyzed "topicid":"tid\_9", // as routing.path, not\_analyzed "rate": 5 // not\_analyzed, for terms facet } and lets find each rate counts for a topic: { "query": { "filtered": { "query": { "term": { "topicid": "tid\_9" // filtering by topicid } } } }, "facets": { "ratings" : { "terms" : { "field" : "rate", "size" : 5, "order" : "term" } } } } thanks in advance -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [April 13, 2013, 7:10am UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/2 "2013-04-13T07:10:23Z")
You can only facet on indexed fields, which means the data is retrieve from the field caches and not source. -- Ivan On Fri, Apr 12, 2013 at 5:31 AM, Ahmet Akyol [liqusha@gmail.com](mailto:liqusha@gmail.com) wrote: > Hi, > > When calculating terms facet, does ES have to look actual data( aka source > which means random access), other than internal index (which may offer > sequential read, I guess). So, my real intention is to understand, how much > random vs sequential access to disk occurs during this type facet > calculation. > > Of course, it depends on docs and query. So here is a simple example: > > a sample doc of many: > > { > "userid":"uid\_1", // as \_id , not\_analyzed > "topicid":"tid\_9", // as routing.path, not\_analyzed > "rate": 5 // not\_analyzed, for terms facet > } > > and lets find each rate counts for a topic: > > { > "query": { > "filtered": { > "query": { > "term": { > "topicid": "tid\_9" // filtering by topicid > } > } > } > }, > "facets": { > "ratings" : { > "terms" : { > "field" : "rate", > "size" : 5, > "order" : "term" > } > } > } > } > > thanks in advance > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ahmet\_Akyol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmet_akyol/32/2389_2.png) [@Ahmet\_Akyol](https://discuss.elastic.co/u/Ahmet_Akyol)\ **Post date:** [April 13, 2013, 11:06am UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/3 "2013-04-13T11:06:58Z")
Thanks Ivan. I used the "calculation path" referring to the internals of ES which I'm not familiar. Here's the example of what I wanted to know from cassandra documentaion: [http://wiki.apache.org/cassandra/ArchitectureInternals](http://wiki.apache.org/cassandra/ArchitectureInternals) As you already know, in NoSql world, "one size fits all" rule doesn't apply and we have to choose the product very carefully for our problems. And knowing the internals even a little bit is very helpful when you decide. Then, you can understand the power and the weakness of the solution. So, back to my question, It'll be nice to see a documentation about "write path" and "read path". On Saturday, April 13, 2013 10:10:23 AM UTC+3, Ivan Brusic wrote: > You can only facet on indexed fields, which means the data is retrieve > from the field caches and not source. > > -- > Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [April 13, 2013, 5:08pm UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/4 "2013-04-13T17:08:06Z")
Disk seeks and data consistency are different issues. Elasticsearch does provide some ability to change write consistency via quorums: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Eventual consistency does not affect me, so I leave the defaults enabled and embrace out-of-order updates. I would not hold my breath waiting for documentation on the internals. 🙂 Cheers, Ivan On Sat, Apr 13, 2013 at 4:06 AM, Ahmet Akyol [liqusha@gmail.com](mailto:liqusha@gmail.com) wrote: > Thanks Ivan. > > I used the "calculation path" referring to the internals of ES which I'm > not familiar. Here's the example of what I wanted to know from cassandra > documentaion: > > [http://wiki.apache.org/cassandra/ArchitectureInternals](http://wiki.apache.org/cassandra/ArchitectureInternals) > > As you already know, in NoSql world, "one size fits all" rule doesn't > apply and we have to choose the product very carefully for our problems. > And knowing the internals even a little bit is very helpful when you > decide. Then, you can understand the power and the weakness of the solution. > > So, back to my question, It'll be nice to see a documentation about "write > path" and "read path". > > On Saturday, April 13, 2013 10:10:23 AM UTC+3, Ivan Brusic wrote: > > > You can only facet on indexed fields, which means the data is retrieve > > from the field caches and not source. > > > > -- > > Ivan > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ahmet\_Akyol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmet_akyol/32/2389_2.png) [@Ahmet\_Akyol](https://discuss.elastic.co/u/Ahmet_Akyol)\ **Post date:** [April 13, 2013, 6:42pm UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/5 "2013-04-13T18:42:16Z")
OK, my bad... Here is a better example documentation: [http://www.datastax.com/docs/1.2/dml/about\_reads](http://www.datastax.com/docs/1.2/dml/about_reads) This page explains how reads work with cassandra at high level. Bloom filters, caches, memtables, SStables etc. But for this topic, what matters is: "Finally, Cassandra performs a single seek and a sequential read of columns (a range read) in the SSTable if the columns are contiguous, and returns the result set." With HBase sequential row access and with Cassandra contiguous column access (wide rows) are the key part of schema design because of sequential reads on disk (SSTable, HFile). Back to question, let's look field caches, as you pointed out: [http://www.elasticsearch.org/guide/reference/index-modules/fielddata/](http://www.elasticsearch.org/guide/reference/index-modules/fielddata/) "The field data cache can be expensive to build for a field, so its recommended to have enough memory to allocate it, and to keep it loaded." This explains expensiveness of memory usage but not the disk access part. So the question is now, where and how does this field data come from. In other words, what is the SSTable of ElasticSearch, meaning what happens to index information (analyzed or not) on disk? What is the structure? Best regards, P.s. : Ivan, you're right, it's a desperate expectation, a "documentation on the internals". Maybe a clever (indeed a clairvoyant) guy can say, "Too much talk. You can't explain it but I see what you really mean.The simple answer to your question is ... " in a few sentences 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [April 15, 2013, 12:53am UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/6 "2013-04-15T00:53:33Z")
Hopefully someone for the elasticsearch team or anyone else that really knows the internals can respond, but in the meantime here is a general overview. Elasticsearch is essentially a distributed version of Lucene. Ultimately all the queried content resides in the indices and segments managed by Lucene. Index data is stored in off-heap memory, which is memory not allocated to the JVM. This allocation of memory is why it is suggested to ran the JVM with 50% of the system memory, since Lucene uses the rest. You can tune Lucene's memory management by changing the underlying filesystem implementation. [1] This memory is managed by the operating system, so data outside the size of the system will be paged to disk. If you want to learn more, look into Lucene. The field data (used for facets and sorting) is allocated in the JVM's heap. Elasticsearch uses Google Guava [2] to manage its caches. The settings are tuneable and basically adjust the Google Guava settings. Too large of field data will result in large garbage collections by the JVM. Cache entry can expire depending on the settings used and the amount of data in the cache. The last piece of the puzzle is the transaction log. Documents indexed in Elasticsearch are first placed into the transaction log. If the write consistency is achieved, then the documents are written to the Lucene segments. The Lucene segments are not distributed between shard/nodes, the transaction log is. Lucene queries work from the indices, but Get requests in Elasticsearch will use the transaction log for real-time queries. It is the transaction log that is perhaps least documented and discussed. There are no major configuration settings (AFAIK), so not much is exposed. You can trace the transaction log code (probably starting from a Get request) to learn more. Essentially Elasticsearch loads as much as possible into memory until it can't. Lucene data is managed by the OS, field cache by the JVM. I probably have already written too much, but have also written too little. Cheers, Ivan [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store/) [2] [GitHub - google/guava: Google core libraries for Java](http://code.google.com/p/guava-libraries/wiki/CachesExplained) On Sat, Apr 13, 2013 at 11:42 AM, Ahmet Akyol [liqusha@gmail.com](mailto:liqusha@gmail.com) wrote: > OK, my bad... Here is a better example documentation: > > [http://www.datastax.com/docs/1.2/dml/about\_reads](http://www.datastax.com/docs/1.2/dml/about_reads) > > This page explains how reads work with cassandra at high level. Bloom > filters, caches, memtables, SStables etc. But for this topic, what matters > is: > > "Finally, Cassandra performs a single seek and a sequential read of > columns (a range read) in the SSTable if the columns are contiguous, and > returns the result set." > > With HBase sequential row access and with Cassandra contiguous column > access (wide rows) are the key part of schema design because of sequential > reads on disk (SSTable, HFile). > > Back to question, let's look field caches, as you pointed out: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/fielddata/) > > "The field data cache can be expensive to build for a field, so its > recommended to have enough memory to allocate it, and to keep it loaded." > > This explains expensiveness of memory usage but not the disk access part. > So the question is now, where and how does this field data come from. In > other words, what is the SSTable of Elasticsearch, meaning what happens to > index information (analyzed or not) on disk? What is the structure? > > Best regards, > > P.s. : Ivan, you're right, it's a desperate expectation, a "documentation > on the internals". Maybe a clever (indeed a clairvoyant) guy can say, "Too > much talk. You can't explain it but I see what you really mean.The simple > answer to your question is ... " in a few sentences 🙂 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ahmet\_Akyol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmet_akyol/32/2389_2.png) [@Ahmet\_Akyol](https://discuss.elastic.co/u/Ahmet_Akyol)\ **Post date:** [April 15, 2013, 8:05am UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/7 "2013-04-15T08:05:57Z")
And the Oscar goes to ... 🙂 Ivan, excellent explanation. You summarized much with little which is extremely hard to do. I really appreciate it. Guava cache is a surprise for me (good for guava, not sure for ES though), but "lucune segments" was the most important pointer for me. Of course, you also underlined the importance (and internals) of memory usage for ES quite well. Thanks again. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:41am UTC](https://discuss.elastic.co/t/terms-facet-calculation-path-random-vs-sequential-disk-access/11564/8 "2017-07-06T02:41:06Z")