# Hadoop as Long-Term Archive for Elasticsearch, but will it work using hadoop compression like gzip? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 10, 2015, 9:11am UTC](https://discuss.elastic.co/t/hadoop-as-long-term-archive-for-elasticsearch-but-will-it-work-using-hadoop-compression-like-gzip/23182 "2015-04-10T09:11:09Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rodrigomerino](https://avatars.discourse-cdn.com/v4/letter/r/f14d63/32.png) [@rodrigomerino](https://discuss.elastic.co/u/rodrigomerino)\ **Post date:** [April 10, 2015, 9:11am UTC](https://discuss.elastic.co/t/hadoop-as-long-term-archive-for-elasticsearch-but-will-it-work-using-hadoop-compression-like-gzip/23182/1 "2015-04-10T09:11:09Z")
Hi all, we are going to receive a _huge amount of logs_ and we are going to process them with the _typical ELK stack_. But because of volume we plan to keep, in ElasticSeach a week or at most a month of logs. After that time we plan to use the Elasticsearch - hadoop integration, mainly for Long-Term Archive, so _older logs are moved to Hadoop_. Disk space is a very important issue here and _we plan to use hadoop compression codecs like gzip_, for those logs older than a month. _If we use compression in hadoop, can we still index + graph data stored in logs older than a month?_ Or for this goal, logs must be stored in a raw format? As far as I have read, if logs (in raw) are stored in hadoop, thanks to the elasticsearch-hadoop integration, elasticsearch can index on them, and kibana can report seamlessly both logs, current logs (stored locally in elasticsearch servers) or older logs (in hadoop). Please correct me if I am wrong. The question is if we lose this feature if we compress the logs in hadoop. Any help is appreciated. Thanks and best regards, Rodrigo. P.S.: Do not hesitate to challenge the architecture/data-flow as well, if you think there are better ways to do it.Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [April 10, 2015, 11:32am UTC](https://discuss.elastic.co/t/hadoop-as-long-term-archive-for-elasticsearch-but-will-it-work-using-hadoop-compression-like-gzip/23182/2 "2015-04-10T11:32:11Z")
Es-Hadoop leverages the existing Hadoop infrastrcture. So whatever compression or splitting is used by your infrastructure, it will simply work with Elasticsearch as well. Take a map reduce job - on the reading part you can use whatever InputFormat you are currently using (to deal with gzip or what have you) and as an OutputFormat the Elasticsearch one. Everything is transparent and works through your existing infrastructure. Notice that you don't need a raw format (what is that) - as long as your data can be read into Hadoop Map/Reduce, Pig, Hive, Cascading, Storm or Spark it can also be written/indexed to Elasticsearch. And vice-versa. On 4/10/15 12:11 PM, Rodrigo Merino wrote: > Hi all, > > we are going to receive a _huge amount of logs_ and we are going to process them with the _typical ELK stack_. But > because of volume we plan to keep, in ElasticSeach a week or at most a month of logs. After that time we plan to use the > Elasticsearch - hadoop integration, mainly for Long-Term Archive, so _older logs are moved to Hadoop_. Disk space is a > very important issue here and _we plan to use hadoop compression codecs like gzip_, for those logs older than a month. > > _If we use compression in hadoop, can we still index + graph data stored in logs older than a month?_ Or for this goal, > logs must be stored in a raw format? > > As far as I have read, if logs (in raw) are stored in hadoop, thanks to the elasticsearch-hadoop integration, > elasticsearch can index on them, and kibana can report seamlessly both logs, current logs (stored locally in > elasticsearch servers) or older logs (in hadoop). Please correct me if I am wrong. The question is if we lose this > feature if we compress the logs in hadoop. > > Any help is appreciated. > > Thanks and best regards, > Rodrigo. > > P.S.: Do not hesitate to challenge the architecture/data-flow as well, if you think there are better ways to do it.Thanks. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5527B4BB.9040409%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/5527B4BB.9040409%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 10, 2015, 11:42am UTC](https://discuss.elastic.co/t/hadoop-as-long-term-archive-for-elasticsearch-but-will-it-work-using-hadoop-compression-like-gzip/23182/3 "2015-04-10T11:42:08Z")
Once it is out of ES you cannot use kibana on it. On 10 April 2015 at 19:11, Rodrigo Merino [rodrigo.merino@gmail.com](mailto:rodrigo.merino@gmail.com) wrote: > Hi all, > > we are going to receive a _huge amount of logs_ and we are going to > process them with the _typical ELK stack_. But because of volume we plan > to keep, in ElasticSeach a week or at most a month of logs. After that time > we plan to use the Elasticsearch - hadoop integration, mainly for Long-Term > Archive, so _older logs are moved to Hadoop_. Disk space is a very > important issue here and _we plan to use hadoop compression codecs like > gzip_, for those logs older than a month. > > _If we use compression in hadoop, can we still index + graph data stored > in logs older than a month?_ Or for this goal, logs must be stored in a > raw format? > > As far as I have read, if logs (in raw) are stored in hadoop, thanks to > the elasticsearch-hadoop integration, elasticsearch can index on them, and > kibana can report seamlessly both logs, current logs (stored locally in > elasticsearch servers) or older logs (in hadoop). Please correct me if I am > wrong. The question is if we lose this feature if we compress the logs in > hadoop. > > Any help is appreciated. > > Thanks and best regards, > Rodrigo. > > P.S.: Do not hesitate to challenge the architecture/data-flow as well, if > you think there are better ways to do it.Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5810a405-8fd8-41f6-b949-9c36cb856b9d%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8PAGQuZzPs6uW-CMPs5mQit78hsrmSJpcOQ\_1a518dog%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8PAGQuZzPs6uW-CMPs5mQit78hsrmSJpcOQ_1a518dog%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:20am UTC](https://discuss.elastic.co/t/hadoop-as-long-term-archive-for-elasticsearch-but-will-it-work-using-hadoop-compression-like-gzip/23182/4 "2017-07-06T00:20:24Z")