I have a question about hourly sharding with either logstash or fluentd.
Since we are, or will be using, a set up called FLEKZ. I am trying to
integrate both logstash and fluentd together, which work well with each
other. However, I have a business requirement for a rolling 24hour shard
deletion.
When I add
logstash_dateformat %Y.%m.%d.%H
in fluentd and
index => "logstash-%{+YYYY.MM.dd.HH}"
into logstash.
Elasticsearch cannot find the indices anymore. I go onto Kibana and they
cannot be found. I switch back to the normal Y.m.d in both and the
information is back on the screen. Using the api I am also not able to
search any of the indices. Is there something I am doing wrong or is there
something in the config file that I am missing?
Did you remember to change the "Timestamping" on Kibana so that it would
know you are using an hourly index ? Go the index configuration screen to
see that.
On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote:
Hello All,
I have a question about hourly sharding with either logstash or fluentd.
Since we are, or will be using, a set up called FLEKZ. I am trying to
integrate both logstash and fluentd together, which work well with each
other. However, I have a business requirement for a rolling 24hour shard
deletion.
When I add
logstash_dateformat %Y.%m.%d.%H
in fluentd and
index => "logstash-%{+YYYY.MM.dd.HH}"
into logstash.
Elasticsearch cannot find the indices anymore. I go onto Kibana and they
cannot be found. I switch back to the normal Y.m.d in both and the
information is back on the screen. Using the api I am also not able to
search any of the indices. Is there something I am doing wrong or is there
something in the config file that I am missing?
On 5 June 2014 02:29, Antonio Augusto Santos mkhaos7@gmail.com wrote:
Hey There,
Did you remember to change the "Timestamping" on Kibana so that it would
know you are using an hourly index ? Go the index configuration screen to
see that.
On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote:
Hello All,
I have a question about hourly sharding with either logstash or fluentd.
Since we are, or will be using, a set up called FLEKZ. I am trying to
integrate both logstash and fluentd together, which work well with each
other. However, I have a business requirement for a rolling 24hour shard
deletion.
When I add
logstash_dateformat %Y.%m.%d.%H
in fluentd and
index => "logstash-%{+YYYY.MM.dd.HH}"
into logstash.
Elasticsearch cannot find the indices anymore. I go onto Kibana and they
cannot be found. I switch back to the normal Y.m.d in both and the
information is back on the screen. Using the api I am also not able to
search any of the indices. Is there something I am doing wrong or is there
something in the config file that I am missing?
What are you calling "lot of resources" ? And how do you go about detecting
it?
Currently I'm ussing ttls for rolling old logs from my cluster. Its pretty
small currently (about 40GB of data), but as its get bigger I want to know
it it will pose a problem.
Thanks
On Wednesday, June 4, 2014 7:46:42 PM UTC-3, Mark Walkom wrote:
TTL isn't the best idea as it consumes a lot of resources. You're better
off getting your hourly indexes working.
On 5 June 2014 02:29, Antonio Augusto Santos <mkh...@gmail.com
<javascript:>> wrote:
Hey There,
Did you remember to change the "Timestamping" on Kibana so that it would
know you are using an hourly index ? Go the index configuration screen to
see that.
On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote:
Hello All,
I have a question about hourly sharding with either logstash or fluentd.
Since we are, or will be using, a set up called FLEKZ. I am trying to
integrate both logstash and fluentd together, which work well with each
other. However, I have a business requirement for a rolling 24hour shard
deletion.
When I add
logstash_dateformat %Y.%m.%d.%H
in fluentd and
index => "logstash-%{+YYYY.MM.dd.HH}"
into logstash.
Elasticsearch cannot find the indices anymore. I go onto Kibana and they
cannot be found. I switch back to the normal Y.m.d in both and the
information is back on the screen. Using the api I am also not able to
search any of the indices. Is there something I am doing wrong or is there
something in the config file that I am missing?
I thought I replied to this yesterday....Anyways it was with kibana. Thank
you for that.
On Wednesday, June 4, 2014 9:29:18 AM UTC-7, Antonio Augusto Santos wrote:
Hey There,
Did you remember to change the "Timestamping" on Kibana so that it would
know you are using an hourly index ? Go the index configuration screen to
see that.
On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote:
Hello All,
I have a question about hourly sharding with either logstash or fluentd.
Since we are, or will be using, a set up called FLEKZ. I am trying to
integrate both logstash and fluentd together, which work well with each
other. However, I have a business requirement for a rolling 24hour shard
deletion.
When I add
logstash_dateformat %Y.%m.%d.%H
in fluentd and
index => "logstash-%{+YYYY.MM.dd.HH}"
into logstash.
Elasticsearch cannot find the indices anymore. I go onto Kibana and they
cannot be found. I switch back to the normal Y.m.d in both and the
information is back on the screen. Using the api I am also not able to
search any of the indices. Is there something I am doing wrong or is there
something in the config file that I am missing?
On 5 June 2014 22:52, Antonio Augusto Santos mkhaos7@gmail.com wrote:
Hey Mark,
What are you calling "lot of resources" ? And how do you go about
detecting it?
Currently I'm ussing ttls for rolling old logs from my cluster. Its pretty
small currently (about 40GB of data), but as its get bigger I want to know
it it will pose a problem.
Thanks
On Wednesday, June 4, 2014 7:46:42 PM UTC-3, Mark Walkom wrote:
TTL isn't the best idea as it consumes a lot of resources. You're better
off getting your hourly indexes working.
On 5 June 2014 02:29, Antonio Augusto Santos mkh...@gmail.com wrote:
Hey There,
Did you remember to change the "Timestamping" on Kibana so that it would
know you are using an hourly index ? Go the index configuration screen to
see that.
Also, if you have the requirement for 24 hour roll out, did you try
enabling _ttl (Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/mapping-ttl-field.html) on your indices ? Like that
the docs older than the specified time would be automatically deleted.
On Wednesday, June 4, 2014 12:16:56 PM UTC-3, Kellan Strong wrote:
Hello All,
I have a question about hourly sharding with either logstash or
fluentd. Since we are, or will be using, a set up called FLEKZ. I am trying
to integrate both logstash and fluentd together, which work well with each
other. However, I have a business requirement for a rolling 24hour shard
deletion.
When I add
logstash_dateformat %Y.%m.%d.%H
in fluentd and
index => "logstash-%{+YYYY.MM.dd.HH}"
into logstash.
Elasticsearch cannot find the indices anymore. I go onto Kibana and
they cannot be found. I switch back to the normal Y.m.d in both and the
information is back on the screen. Using the api I am also not able to
search any of the indices. Is there something I am doing wrong or is there
something in the config file that I am missing?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.