Is elasticsearch a good solution for storing logs for some time?
I was asking around on the logstash irc channel and the general consensus
is that elasticsearch is not storage friendly.
Does somoene her have any experience with this?
I want it to store log messages.
Sure you can use ES or Solr or even straight Lucene for long-term log
storage. Just make sure you shard well if you want to keep searching
it efficiently and replicate sufficiently not to lose data.
Otis
Sematext is Hiring Search People World-Wide -- Jobs - Sematext
On Dec 18, 10:06 pm, GambitK alfred.rap...@gmail.com wrote:
Is elasticsearch a good solution for storing logs for some time?
I was asking around on the logstash irc channel and the general consensus
is that elasticsearch is not storage friendly.
Does somoene her have any experience with this?
I want it to store log messages.
What do you mean by storage friendly? You can store your data there, sure.
On Mon, Dec 19, 2011 at 5:06 AM, GambitK alfred.rapozo@gmail.com wrote:
Is elasticsearch a good solution for storing logs for some time?
I was asking around on the logstash irc channel and the general consensus
is that elasticsearch is not storage friendly.
Does somoene her have any experience with this?
I want it to store log messages.
Hi Gambitk,
I coded a Java implementation of SLF4J that index application logs in
ES over zeromq.
I used it on few small apps, it works well. Maybe it will help you.
Tanguy
On 19 déc, 04:06, GambitK alfred.rap...@gmail.com wrote:
Is elasticsearch a good solution for storing logs for some time?
I was asking around on the logstash irc channel and the general consensus
is that elasticsearch is not storage friendly.
Does somoene her have any experience with this?
I want it to store log messages.
Sorry for not being clear.
I meant in terms of disk space requirements. I asked about 3 months of logs
and someone mentioned 3TB or something of that sort, which I find a lot of
space for that time frame.
Again, hard to answer without knowing what log file you index and how much
"raw" size you have. You can't get away from the actual inverted index
"size", and storing the actual _source is just additional "bytes", which
you can compress if you want.
On Mon, Dec 19, 2011 at 12:53 PM, GambitK alfred.rapozo@gmail.com wrote:
Sorry for not being clear.
I meant in terms of disk space requirements. I asked about 3 months of
logs and someone mentioned 3TB or something of that sort, which I find a
lot of space for that time frame.
Sources are syslog format logs, comming from firewalls, windows servers,
routers, etc, about 1GB a day.
The idea is to have a couple of weeks worth of logs indexed to be able to
search it.
well if you have 1GB a day then you'll have a similar big index
(again: depending on your requirements as Shay noted).
So its not 3TB its more about 100-500GB in your case. But the limiting
factor will be RAM or how many days you want that are searchable ...
Peter.
On 20 Dez., 19:12, GambitK alfred.rap...@gmail.com wrote:
Sources are syslog format logs, comming from firewalls, windows servers,
routers, etc, about 1GB a day.
The idea is to have a couple of weeks worth of logs indexed to be able to
search it.