I am stuck between using elasticsearch and cassandra for log data. Most of
the log data that I will have is not full text search based, most of it is
aggrated data and information about the logs them self. The reason that I
am looking at something like full text serach over something like mongodb
is at my day job we use mongodb and it has been a love heat sort of setup,
I dont want to have to deal with all the node types I like the simplicity
of sharded nothing systems. What I dont know is if elasticsearch is going
to be okay that most of the data is not full text search like, i may input
some raw logs for real time log stuff but it would only live for a set
period of time.
Elasticsearch is a good choice for this. Lots of people are using ES for
log indexing. Most of the time they search on 1 or more indexed fields.
If you will need to simply retrieve big block of logs, optionally doing
some processing of it, then Cassandra (Or HBase!) are good choices. If you
think you'll need to look up logs by anything other than some primary key,
then ES is a better choice.
Otis
Solr & Elasticsearch Consulting
On Thursday, December 20, 2012 6:27:54 AM UTC-5, Wojons Tech wrote:
I am stuck between using elasticsearch and cassandra for log data. Most of
the log data that I will have is not full text search based, most of it is
aggrated data and information about the logs them self. The reason that I
am looking at something like full text serach over something like mongodb
is at my day job we use mongodb and it has been a love heat sort of setup,
I dont want to have to deal with all the node types I like the simplicity
of sharded nothing systems. What I dont know is if elasticsearch is going
to be okay that most of the data is not full text search like, i may input
some raw logs for real time log stuff but it would only live for a set
period of time.
Thank you, would you also belive that if i use it to store metrics on the
data that it is also a good choice?
On Thursday, December 20, 2012 8:20:09 PM UTC-8, Otis Gospodnetic wrote:
Hi,
Elasticsearch is a good choice for this. Lots of people are using ES for
log indexing. Most of the time they search on 1 or more indexed fields.
If you will need to simply retrieve big block of logs, optionally doing
some processing of it, then Cassandra (Or HBase!) are good choices. If you
think you'll need to look up logs by anything other than some primary key,
then ES is a better choice.
On Thursday, December 20, 2012 6:27:54 AM UTC-5, Wojons Tech wrote:
I am stuck between using elasticsearch and cassandra for log data. Most
of the log data that I will have is not full text search based, most of it
is aggrated data and information about the logs them self. The reason that
I am looking at something like full text serach over something like mongodb
is at my day job we use mongodb and it has been a love heat sort of setup,
I dont want to have to deal with all the node types I like the simplicity
of sharded nothing systems. What I dont know is if elasticsearch is going
to be okay that most of the data is not full text search like, i may input
some raw logs for real time log stuff but it would only live for a set
period of time.
What I dont know is if elasticsearch is going to be okay that most of the
data is not full text search like, i may input some raw logs for real time
log stuff but it would only live for a set period of time.
That is a pretty nice use case for Elasticsearch, one for which it is
suited very well. Look at the http://logstash.net and http://kibana.org
to get inspiration and play around.
I would like to add to that, that I'm also working on such a project. It's
entirely managed, so no management/knowledge needed. If you need any
details, or would like to discuss some methods we use, feel free to contact
me.
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
What I dont know is if elasticsearch is going to be okay that most of the
data is not full text search like, i may input some raw logs for real time
log stuff but it would only live for a set period of time.
That is a pretty nice use case for Elasticsearch, one for which it is
suited very well. Look at the http://logstash.net and http://kibana.org
to get inspiration and play around.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.