# Hourly Shards Elasticsearch/Kibana **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2014, 3:16pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908 "2014-06-04T15:16:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Kellan\_Strong](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@Kellan\_Strong](https://discuss.elastic.co/u/Kellan_Strong)\ **Post date:** [June 4, 2014, 3:16pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/1 "2014-06-04T15:16:56Z")
Hello All, I have a question about hourly sharding with either logstash or fluentd. Since we are, or will be using, a set up called FLEKZ. I am trying to integrate both logstash and fluentd together, which work well with each other. However, I have a business requirement for a rolling 24hour shard deletion. When I add logstash\_dateformat %Y.%m.%d.%H in fluentd and index =\> "logstash-%{+YYYY.MM.dd.HH}" into logstash. Elasticsearch cannot find the indices anymore. I go onto Kibana and they cannot be found. I switch back to the normal Y.m.d in both and the information is back on the screen. Using the api I am also not able to search any of the indices. Is there something I am doing wrong or is there something in the config file that I am missing? Thank you for your help, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a0dc08e6-c570-4305-bc0b-808937551f54%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a0dc08e6-c570-4305-bc0b-808937551f54%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Antonio\_Augusto\_Sant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonio_augusto_sant/32/82851_2.png) [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)\ **Post date:** [June 4, 2014, 4:29pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/2 "2014-06-04T16:29:18Z")
Hey There, Did you remember to change the "Timestamping" on Kibana so that it would know you are using an hourly index ? Go the index configuration screen to see that. Also, if you have the requirement for 24 hour roll out, did you try enabling \_ttl ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)) on your indices ? Like that the docs older than the specified time would be automatically deleted. On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote: > Hello All, > > I have a question about hourly sharding with either logstash or fluentd. > Since we are, or will be using, a set up called FLEKZ. I am trying to > integrate both logstash and fluentd together, which work well with each > other. However, I have a business requirement for a rolling 24hour shard > deletion. > > When I add > > logstash\_dateformat %Y.%m.%d.%H > > in fluentd and > > index =\> "logstash-%{+YYYY.MM.dd.HH}" > > into logstash. > > Elasticsearch cannot find the indices anymore. I go onto Kibana and they > cannot be found. I switch back to the normal Y.m.d in both and the > information is back on the screen. Using the api I am also not able to > search any of the indices. Is there something I am doing wrong or is there > something in the config file that I am missing? > > Thank you for your help, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 4, 2014, 10:46pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/3 "2014-06-04T22:46:04Z")
TTL isn't the best idea as it consumes a lot of resources. You're better off getting your hourly indexes working. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 5 June 2014 02:29, Antonio Augusto Santos [mkhaos7@gmail.com](mailto:mkhaos7@gmail.com) wrote: > Hey There, > > Did you remember to change the "Timestamping" on Kibana so that it would > know you are using an hourly index ? Go the index configuration screen to > see that. > > Also, if you have the requirement for 24 hour roll out, did you try > enabling \_ttl ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)) > on your indices ? Like that the docs older than the specified time would be > automatically deleted. > > On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote: > > > Hello All, > > > > I have a question about hourly sharding with either logstash or fluentd. > > Since we are, or will be using, a set up called FLEKZ. I am trying to > > integrate both logstash and fluentd together, which work well with each > > other. However, I have a business requirement for a rolling 24hour shard > > deletion. > > > > When I add > > > > logstash\_dateformat %Y.%m.%d.%H > > > > in fluentd and > > > > index =\> "logstash-%{+YYYY.MM.dd.HH}" > > > > into logstash. > > > > Elasticsearch cannot find the indices anymore. I go onto Kibana and they > > cannot be found. I switch back to the normal Y.m.d in both and the > > information is back on the screen. Using the api I am also not able to > > search any of the indices. Is there something I am doing wrong or is there > > something in the config file that I am missing? > > > > Thank you for your help, > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Zp%2BsoR-uUSDmVm\_5MRBZs4AHBh-9ggT9twK5ucR\_vRyg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Zp%2BsoR-uUSDmVm_5MRBZs4AHBh-9ggT9twK5ucR_vRyg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Antonio\_Augusto\_Sant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonio_augusto_sant/32/82851_2.png) [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)\ **Post date:** [June 5, 2014, 12:52pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/4 "2014-06-05T12:52:08Z")
Hey Mark, What are you calling "lot of resources" ? And how do you go about detecting it? Currently I'm ussing ttls for rolling old logs from my cluster. Its pretty small currently (about 40GB of data), but as its get bigger I want to know it it will pose a problem. Thanks On Wednesday, June 4, 2014 7:46:42 PM UTC-3, Mark Walkom wrote: > TTL isn't the best idea as it consumes a lot of resources. You're better > off getting your hourly indexes working. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 5 June 2014 02:29, Antonio Augusto Santos \<[mkh...@gmail.com](mailto:mkh...@gmail.com) > \\> wrote: > > > Hey There, > > > > Did you remember to change the "Timestamping" on Kibana so that it would > > know you are using an hourly index ? Go the index configuration screen to > > see that. > > > > Also, if you have the requirement for 24 hour roll out, did you try > > enabling \_ttl ( > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)) > > on your indices ? Like that the docs older than the specified time would be > > automatically deleted. > > > > On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote: > > > > > Hello All, > > > > > > I have a question about hourly sharding with either logstash or fluentd. > > > Since we are, or will be using, a set up called FLEKZ. I am trying to > > > integrate both logstash and fluentd together, which work well with each > > > other. However, I have a business requirement for a rolling 24hour shard > > > deletion. > > > > > > When I add > > > > > > logstash\_dateformat %Y.%m.%d.%H > > > > > > in fluentd and > > > > > > index =\> "logstash-%{+YYYY.MM.dd.HH}" > > > > > > into logstash. > > > > > > Elasticsearch cannot find the indices anymore. I go onto Kibana and they > > > cannot be found. I switch back to the normal Y.m.d in both and the > > > information is back on the screen. Using the api I am also not able to > > > search any of the indices. Is there something I am doing wrong or is there > > > something in the config file that I am missing? > > > > > > Thank you for your help, > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6316097e-06e8-4fb2-97fb-deb532d1893f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6316097e-06e8-4fb2-97fb-deb532d1893f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Kellan\_Strong](https://avatars.discourse-cdn.com/v4/letter/k/7993a0/32.png) [@Kellan\_Strong](https://discuss.elastic.co/u/Kellan_Strong)\ **Post date:** [June 5, 2014, 4:03pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/5 "2014-06-05T16:03:18Z")
I thought I replied to this yesterday....Anyways it was with kibana. Thank you for that. On Wednesday, June 4, 2014 9:29:18 AM UTC-7, Antonio Augusto Santos wrote: > Hey There, > > Did you remember to change the "Timestamping" on Kibana so that it would > know you are using an hourly index ? Go the index configuration screen to > see that. > > Also, if you have the requirement for 24 hour roll out, did you try > enabling \_ttl ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html)) > on your indices ? Like that the docs older than the specified time would be > automatically deleted. > > On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote: > > > Hello All, > > > > I have a question about hourly sharding with either logstash or fluentd. > > Since we are, or will be using, a set up called FLEKZ. I am trying to > > integrate both logstash and fluentd together, which work well with each > > other. However, I have a business requirement for a rolling 24hour shard > > deletion. > > > > When I add > > > > logstash\_dateformat %Y.%m.%d.%H > > > > in fluentd and > > > > index =\> "logstash-%{+YYYY.MM.dd.HH}" > > > > into logstash. > > > > Elasticsearch cannot find the indices anymore. I go onto Kibana and they > > cannot be found. I switch back to the normal Y.m.d in both and the > > information is back on the screen. Using the api I am also not able to > > search any of the indices. Is there something I am doing wrong or is there > > something in the config file that I am missing? > > > > Thank you for your help, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/00e45fad-5b07-4cd8-82ca-45dd25d1c71f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00e45fad-5b07-4cd8-82ca-45dd25d1c71f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 5, 2014, 11:21pm UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/6 "2014-06-05T23:21:31Z")
It depends on a few factors, document size, index size, etc etc. If you are using ES for logging data, then best practise is to use timestamped indexes and then just drop old ones as needed using curator. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 5 June 2014 22:52, Antonio Augusto Santos [mkhaos7@gmail.com](mailto:mkhaos7@gmail.com) wrote: > Hey Mark, > > What are you calling "lot of resources" ? And how do you go about > detecting it? > Currently I'm ussing ttls for rolling old logs from my cluster. Its pretty > small currently (about 40GB of data), but as its get bigger I want to know > it it will pose a problem. > > Thanks > > On Wednesday, June 4, 2014 7:46:42 PM UTC-3, Mark Walkom wrote: > > > TTL isn't the best idea as it consumes a lot of resources. You're better > > off getting your hourly indexes working. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 5 June 2014 02:29, Antonio Augusto Santos [mkh...@gmail.com](mailto:mkh...@gmail.com) wrote: > > > > > Hey There, > > > > > > Did you remember to change the "Timestamping" on Kibana so that it would > > > know you are using an hourly index ? Go the index configuration screen to > > > see that. > > > > > > Also, if you have the requirement for 24 hour roll out, did you try > > > enabling \_ttl ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/) > > > reference/current/mapping-ttl-field.html) on your indices ? Like that > > > the docs older than the specified time would be automatically deleted. > > > > > > On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote: > > > > > > > Hello All, > > > > > > > > I have a question about hourly sharding with either logstash or > > > > fluentd. Since we are, or will be using, a set up called FLEKZ. I am trying > > > > to integrate both logstash and fluentd together, which work well with each > > > > other. However, I have a business requirement for a rolling 24hour shard > > > > deletion. > > > > > > > > When I add > > > > > > > > logstash\_dateformat %Y.%m.%d.%H > > > > > > > > in fluentd and > > > > > > > > index =\> "logstash-%{+YYYY.MM.dd.HH}" > > > > > > > > into logstash. > > > > > > > > Elasticsearch cannot find the indices anymore. I go onto Kibana and > > > > they cannot be found. I switch back to the normal Y.m.d in both and the > > > > information is back on the screen. Using the api I am also not able to > > > > search any of the indices. Is there something I am doing wrong or is there > > > > something in the config file that I am missing? > > > > > > > > Thank you for your help, > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a1bceabb-ea26-4aa5-8358-92f6f8e2ae1e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/6316097e-06e8-4fb2-97fb-deb532d1893f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6316097e-06e8-4fb2-97fb-deb532d1893f%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/6316097e-06e8-4fb2-97fb-deb532d1893f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6316097e-06e8-4fb2-97fb-deb532d1893f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZqHS\_op963EkQpSBbhqiamuaLLAurMG55HbFmxN-Gydg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZqHS_op963EkQpSBbhqiamuaLLAurMG55HbFmxN-Gydg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:24am UTC](https://discuss.elastic.co/t/hourly-shards-elasticsearch-kibana/17908/7 "2017-07-06T01:24:19Z")