# What is better - create several document types or several indices? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 11, 2014, 2:09am UTC](https://discuss.elastic.co/t/what-is-better-create-several-document-types-or-several-indices/19728 "2014-09-11T02:09:08Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [September 11, 2014, 2:09am UTC](https://discuss.elastic.co/t/what-is-better-create-several-document-types-or-several-indices/19728/1 "2014-09-11T02:09:08Z")
We use Elasticsearch to aggregate several types of logs - web server logs, application logs, windows event logs, statistics, etc. As far as I understand I can do one of the following: 1, Send each log to its own index and when I need to combine them in query - specify several indices in Kibana settings; 1. Send all logs to the same index (we turn them over every day) and give logs from different sources different document types; 2. Do more or less nothing, push all documents together without distinguishing them explicitly; My question is - what are advantages and disadvantages of each approach? We have substantial amount of logs going in every second, but querying is rather rare, at least so far. Thank you! Konstantin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [September 11, 2014, 4:26am UTC](https://discuss.elastic.co/t/what-is-better-create-several-document-types-or-several-indices/19728/2 "2014-09-11T04:26:58Z")
Hello , My advice would be to keep all the logs in a single index , but apply index tailing. That is write logs of a day or hour ( depending upon traffic) to each index like logstash does. So name of the index would be of format logs-`yyyy-MM-dd` This way , you wont be stuck with the fixed shard problem and dynamic horizontal scaling can be achieved. Also , it would be a wise idea to remove old logs using TTL facility OR closing old index or even take a snapshot and remove the index. TTL - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Index Close - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Thanks Vineeth On Thu, Sep 11, 2014 at 7:39 AM, Konstantin Erman [konste@gmail.com](mailto:konste@gmail.com) wrote: > We use Elasticsearch to aggregate several types of logs - web server logs, > application logs, windows event logs, statistics, etc. > > As far as I understand I can do one of the following: > 1, Send each log to its own index and when I need to combine them in query > > - specify several indices in Kibana settings; > > 1. Send all logs to the same index (we turn them over every day) and give > logs from different sources different document types; > 2. Do more or less nothing, push all documents together without > distinguishing them explicitly; > > My question is - what are advantages and disadvantages of each approach? > We have substantial amount of logs going in every second, but querying is > rather rare, at least so far. > > Thank you! > Konstantin > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kmDEJ%2BmhfX8RtGm9KAiBKEK%3DT1-1r3kj7pCnNNwMY-PA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kmDEJ%2BmhfX8RtGm9KAiBKEK%3DT1-1r3kj7pCnNNwMY-PA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Konstantin\_Erman](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@Konstantin\_Erman](https://discuss.elastic.co/u/Konstantin_Erman)\ **Post date:** [September 11, 2014, 12:57pm UTC](https://discuss.elastic.co/t/what-is-better-create-several-document-types-or-several-indices/19728/3 "2014-09-11T12:57:11Z")
Vineeth, thank you for your advice! It seems we already do everything as you said. We name indices in Logstash style with the date. Is that what you are referring to as tailing? Creating an index per hour would lead to hundreds of indices open. I wonder what are the guidelines regarding the number of indices vs their size? We also close less interesting logs and in a couple weeks delete them. BUT STILL, with all that in place my original question still stands: different document types in the same index or rather different indices for different document types. What are the rules of thumb? Konstantin On Wednesday, September 10, 2014 9:27:05 PM UTC-7, vineeth mohan wrote: > Hello , > > My advice would be to keep all the logs in a single index , but apply > index tailing. > That is write logs of a day or hour ( depending upon traffic) to each > index like logstash does. > So name of the index would be of format logs-`yyyy-MM-dd` > This way , you wont be stuck with the fixed shard problem and dynamic > horizontal scaling can be achieved. > Also , it would be a wise idea to remove old logs using TTL facility OR > closing old index or even take a snapshot and remove the index. > > TTL - > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#index-ttl) > Index Close - > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html#indices-open-close) > > Thanks > Vineeth > > On Thu, Sep 11, 2014 at 7:39 AM, Konstantin Erman \<[kon...@gmail.com](mailto:kon...@gmail.com) > \\> wrote: > > > We use Elasticsearch to aggregate several types of logs - web server > > logs, application logs, windows event logs, statistics, etc. > > > > As far as I understand I can do one of the following: > > 1, Send each log to its own index and when I need to combine them in > > query - specify several indices in Kibana settings; > > 2. Send all logs to the same index (we turn them over every day) and give > > logs from different sources different document types; > > 3. Do more or less nothing, push all documents together without > > distinguishing them explicitly; > > > > My question is - what are advantages and disadvantages of each approach? > > We have substantial amount of logs going in every second, but querying is > > rather rare, at least so far. > > > > Thank you! > > Konstantin > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e41e4959-6a45-417a-8ba6-856abcd33350%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5df677a5-46d9-4ecd-9bb9-a82f7897753b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5df677a5-46d9-4ecd-9bb9-a82f7897753b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\ **Post date:** [September 11, 2014, 1:21pm UTC](https://discuss.elastic.co/t/what-is-better-create-several-document-types-or-several-indices/19728/4 "2014-09-11T13:21:04Z")
Every index has a minimum of one shard. Multiple types can live in the same shard. Shards both have maintenance overheads and slow down queries. However, if you have a lot of targeted queries you can more easily reduce the shards accessed by reducing indexes than you could if you had multi-tenancy. I could be missing something but I don't think you can have multiple routing values in a query, but someone may want to query multiple log types. So it depends. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ed078e55-b9e3-4c82-8285-a08ba5f90e21%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ed078e55-b9e3-4c82-8285-a08ba5f90e21%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:03am UTC](https://discuss.elastic.co/t/what-is-better-create-several-document-types-or-several-indices/19728/5 "2017-07-06T01:03:06Z")