# Questions TTL/GET/ **URL:** \ **Category:** Elasticsearch\ **Created:** [September 17, 2013, 5:05pm UTC](https://discuss.elastic.co/t/questions-ttl-get/13652 "2013-09-17T17:05:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [September 17, 2013, 5:05pm UTC](https://discuss.elastic.co/t/questions-ttl-get/13652/1 "2013-09-17T17:05:56Z")
How are Get and Multiget queries fetched? What data structure is used to hold this data? Are expired documents indexed and deleted on cron(like mongodb) or are they deleted when segments are merged (cassandra/bigtable)? And what should i use if i want to just filter documents (no scoring), filter, filtered query what? Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [September 17, 2013, 5:22pm UTC](https://discuss.elastic.co/t/questions-ttl-get/13652/2 "2013-09-17T17:22:49Z")
1. I don't fully understand your question. Elasticsearch has a binary serialization protocol - the document is retrieved from the shard, serialized across the network to the coordinating node, and then sent to the client as JSON 2. The TTL cycle runs every 60s by default. Expired documents are marked as deleted and removed during merges. Lucene has no concept of random-access writes - segments are immutable. So the only way to delete a document is to mark it as deleted and remove it during a subsequent merge. This is also why TTL is generally not recommended for most scenarios...it is much slower than simply deleting a time-based index 3. Filtered\_query. The top-level filter should only be used when working with facets and you require fine-grained filter control of search vs. facet results -Zach On Tuesday, September 17, 2013 1:05:56 PM UTC-4, ddorian43 wrote: > How are Get and Multiget queries fetched? What data structure is used to > hold this data? > > Are expired documents indexed and deleted on cron(like mongodb) or are > they deleted when segments are merged (cassandra/bigtable)? > > And what should i use if i want to just filter documents (no scoring), > filter, filtered query what? > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [September 20, 2013, 7:45pm UTC](https://discuss.elastic.co/t/questions-ttl-get/13652/3 "2013-09-20T19:45:07Z")
What i mean: What i mean is how is \_uid indexed? Reversed index? In mongodb TTL, the timestamp is indexed and the db runs a query every x seconds to find the expired documents and delete . While in Cassandra, expired cells are only deleted on compactions. There is not another index that scans like mongodb. Which TTL type does elasticsearch support? It looks like you can't range\_query on \_id field without indexing ? (like get and multiget doesn't require indexing) Also when documents are filtered and no sorting is specified, are documents sorted by \_id ? Thanks On Tuesday, September 17, 2013 7:05:56 PM UTC+2, ddorian43 wrote: > How are Get and Multiget queries fetched? What data structure is used to > hold this data? > > Are expired documents indexed and deleted on cron(like mongodb) or are > they deleted when segments are merged (cassandra/bigtable)? > > And what should i use if i want to just filter documents (no scoring), > filter, filtered query what? > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [September 20, 2013, 8:17pm UTC](https://discuss.elastic.co/t/questions-ttl-get/13652/4 "2013-09-20T20:17:31Z")
Answers inline: What i mean is how is \_uid indexed? Reversed index? > \_uids are indexed into a bloom filter. The process looks like this: 1. Find the appropriate shard via routing, forward request to a node with the shard 2. Sequentially iterate over each segment in the shard 1. Check bloom filter if DocID exists in this segment. If false, move to next shard (blooms guarantee negative results) 2. If yes, perform search through segment, as blooms have some amount of false positive In mongodb TTL, the timestamp is indexed and the db runs a query every x > seconds to find the expired documents and delete . > While in Cassandra, expired cells are only deleted on compactions. There > is not another index that scans like mongodb. Which TTL type does elasticsearch support? See my answer in the first email. Elasticsearch's background process runs every 60s, compares timestamps and then marks documents as deleted. The physical deletion of data doesn't happen until a merge later removes it. It looks like you can't range\_query on \_id field without indexing ? (like > get and multiget doesn't require indexing) Correct. Queries only operate on inverted index data, so they require the \_id to be indexed along with the other fields. Get/Multiget use a different mechanism (detailed above w/ bloom filter), they aren't queries, so they don't require an inverted index. Also when documents are filtered and no sorting is specified, are documents > sorted by \_id ? If docs are filtered and have no score, the score is automatically set to 1. Since all documents have the same score, the resulting order is effectively random. In actuality it is the order of the documents in the segments, but the merge process tends to shuffle them enough that you can consider it random. Hope that helps! -Zach > Thanks > > On Tuesday, September 17, 2013 7:05:56 PM UTC+2, ddorian43 wrote: > > > How are Get and Multiget queries fetched? What data structure is used to > > hold this data? > > > > Are expired documents indexed and deleted on cron(like mongodb) or are > > they deleted when segments are merged (cassandra/bigtable)? > > > > And what should i use if i want to just filter documents (no scoring), > > filter, filtered query what? > > > > Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [October 15, 2013, 3:00pm UTC](https://discuss.elastic.co/t/questions-ttl-get/13652/5 "2013-10-15T15:00:20Z")
Everything helps. 1. If i make a filter + sort, only the filtered\_documents.sorting\_value must reside in memory or all documents?(i'm guessing all) 2. Is the doc\_type saved as a string or some kind of internal id in the \_uid field (so if i have many small documents i should keep this short)? 3. If i disable the "\_all" field(because i don't want to duplicate indexes), how i can simulate it (by searching on all the indexed fields?) ? 4. If i want to use autocomplete on tags field, should i use ngrams or suggester (when starting to type show a list of tags that are like the characters, looks like suggester)? 5. Looks like suggester doesn't return documents ? 6. If i use a field.type(number), and i wont do range queries(only equality), what precision\_step should i use for this field (0 or 1?) ? On Fri, Sep 20, 2013 at 10:17 PM, Zachary Tong [zacharyjtong@gmail.com](mailto:zacharyjtong@gmail.com)wrote: > Answers inline: > > What i mean is how is \_uid indexed? Reversed index? > > > > > \_uids are indexed into a bloom filter. The process looks like this: > > 1. Find the appropriate shard via routing, forward request to a node > with the shard > 2. Sequentially iterate over each segment in the shard > 1. Check bloom filter if DocID exists in this segment. If false, > move to next shard (blooms guarantee negative results) > 2. If yes, perform search through segment, as blooms have some > amount of false positive > > In mongodb TTL, the timestamp is indexed and the db runs a query every x > > > seconds to find the expired documents and delete . > > While in Cassandra, expired cells are only deleted on compactions. There > > is not another index that scans like mongodb. > > Which TTL type does elasticsearch support? > > See my answer in the first email. Elasticsearch's background process runs > every 60s, compares timestamps and then marks documents as deleted. The > physical deletion of data doesn't happen until a merge later removes it. > > It looks like you can't range\_query on \_id field without indexing ? (like > > > get and multiget doesn't require indexing) > > Correct. Queries only operate on inverted index data, so they require the > \_id to be indexed along with the other fields. Get/Multiget use a > different mechanism (detailed above w/ bloom filter), they aren't queries, > so they don't require an inverted index. > > Also when documents are filtered and no sorting is specified, are > > > documents sorted by \_id ? > > If docs are filtered and have no score, the score is automatically set to > > 1. Since all documents have the same score, the resulting order is > effectively random. In actuality it is the order of the documents in the > segments, but the merge process tends to shuffle them enough that you can > consider it random. > > Hope that helps! > -Zach > > > Thanks > > > > On Tuesday, September 17, 2013 7:05:56 PM UTC+2, ddorian43 wrote: > > > > > How are Get and Multiget queries fetched? What data structure is used to > > > hold this data? > > > > > > Are expired documents indexed and deleted on cron(like mongodb) or are > > > they deleted when segments are merged (cassandra/bigtable)? > > > > > > And what should i use if i want to just filter documents (no scoring), > > > filter, filtered query what? > > > > > > Thanks > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/DnyEVBUOv5U/unsubscribe](https://groups.google.com/d/topic/elasticsearch/DnyEVBUOv5U/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:12am UTC](https://discuss.elastic.co/t/questions-ttl-get/13652/6 "2017-07-06T02:12:22Z")