# Elasticsearch for mid/long term storage **URL:** \ **Category:** Elasticsearch\ **Created:** [December 19, 2011, 3:06am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190 "2011-12-19T03:06:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![GambitK](https://avatars.discourse-cdn.com/v4/letter/g/47e85d/32.png) [@GambitK](https://discuss.elastic.co/u/GambitK)\ **Post date:** [December 19, 2011, 3:06am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/1 "2011-12-19T03:06:18Z")
Is elasticsearch a good solution for storing logs for some time? I was asking around on the logstash irc channel and the general consensus is that elasticsearch is not storage friendly. Does somoene her have any experience with this? I want it to store log messages. ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [December 19, 2011, 5:28am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/2 "2011-12-19T05:28:10Z")
Sure you can use ES or Solr or even straight Lucene for long-term log storage. Just make sure you shard well if you want to keep searching it efficiently and replicate sufficiently not to lose data. ## Otis Sematext is Hiring Search People World-Wide -- [Jobs](http://sematext.com/about/jobs.html) On Dec 18, 10:06 pm, GambitK [alfred.rap...@gmail.com](mailto:alfred.rap...@gmail.com) wrote: > Is elasticsearch a good solution for storing logs for some time? > I was asking around on the logstash irc channel and the general consensus > is that elasticsearch is not storage friendly. > Does somoene her have any experience with this? > I want it to store log messages. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 19, 2011, 6:49am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/3 "2011-12-19T06:49:00Z")
What do you mean by storage friendly? You can store your data there, sure. On Mon, Dec 19, 2011 at 5:06 AM, GambitK [alfred.rapozo@gmail.com](mailto:alfred.rapozo@gmail.com) wrote: > Is elasticsearch a good solution for storing logs for some time? > I was asking around on the logstash irc channel and the general consensus > is that elasticsearch is not storage friendly. > Does somoene her have any experience with this? > I want it to store log messages. ---
**Author:** ![Tanguy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguy1/32/1749_2.png) [@Tanguy1](https://discuss.elastic.co/u/Tanguy1)\ **Post date:** [December 19, 2011, 8:10am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/4 "2011-12-19T08:10:27Z")
Hi Gambitk, I coded a Java implementation of SLF4J that index application logs in ES over zeromq. > **[GitHub - tlrx/slf4j-logback-zeromq: A Logback appender which sends ØMQ message](https://github.com/tlrx/slf4j-logback-zeromq)** > > A Logback appender which sends ØMQ message. Contribute to tlrx/slf4j-logback-zeromq development by creating an account on GitHub. I used it on few small apps, it works well. Maybe it will help you. Tanguy On 19 déc, 04:06, GambitK [alfred.rap...@gmail.com](mailto:alfred.rap...@gmail.com) wrote: > Is elasticsearch a good solution for storing logs for some time? > I was asking around on the logstash irc channel and the general consensus > is that elasticsearch is not storage friendly. > Does somoene her have any experience with this? > I want it to store log messages. ---
**Author:** ![GambitK](https://avatars.discourse-cdn.com/v4/letter/g/47e85d/32.png) [@GambitK](https://discuss.elastic.co/u/GambitK)\ **Post date:** [December 19, 2011, 10:53am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/5 "2011-12-19T10:53:07Z")
Sorry for not being clear. I meant in terms of disk space requirements. I asked about 3 months of logs and someone mentioned 3TB or something of that sort, which I find a lot of space for that time frame. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 20, 2011, 3:06pm UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/6 "2011-12-20T15:06:08Z")
Again, hard to answer without knowing what log file you index and how much "raw" size you have. You can't get away from the actual inverted index "size", and storing the actual \_source is just additional "bytes", which you can compress if you want. On Mon, Dec 19, 2011 at 12:53 PM, GambitK [alfred.rapozo@gmail.com](mailto:alfred.rapozo@gmail.com) wrote: > Sorry for not being clear. > I meant in terms of disk space requirements. I asked about 3 months of > logs and someone mentioned 3TB or something of that sort, which I find a > lot of space for that time frame. ---
**Author:** ![GambitK](https://avatars.discourse-cdn.com/v4/letter/g/47e85d/32.png) [@GambitK](https://discuss.elastic.co/u/GambitK)\ **Post date:** [December 20, 2011, 6:12pm UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/7 "2011-12-20T18:12:16Z")
Sources are syslog format logs, comming from firewalls, windows servers, routers, etc, about 1GB a day. The idea is to have a couple of weeks worth of logs indexed to be able to search it. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [December 20, 2011, 7:30pm UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/8 "2011-12-20T19:30:49Z")
well if you have 1GB a day then you'll have a similar big index (again: depending on your requirements as Shay noted). So its not 3TB its more about 100-500GB in your case. But the limiting factor will be RAM or how many days you want that are searchable ... Peter. On 20 Dez., 19:12, GambitK [alfred.rap...@gmail.com](mailto:alfred.rap...@gmail.com) wrote: > Sources are syslog format logs, comming from firewalls, windows servers, > routers, etc, about 1GB a day. > The idea is to have a couple of weeks worth of logs indexed to be able to > search it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:44am UTC](https://discuss.elastic.co/t/elasticsearch-for-mid-long-term-storage/6190/9 "2017-07-06T03:44:51Z")