Need some advice to build a log central

Elastic Stack Elasticsearch
1

Hi All,
I am going to build a log central using ElasticSearch.
I need some advice from anyone who have built it already.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5af88543-5806-4021-83a5-41abc5b2bed6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hello Sang ,

As this is a question answer forum , we highly recommend you to take a shot
yourself and post questions if you have hit a dead end.

Thanks
Vineeth

On Mon, Aug 25, 2014 at 7:56 AM, Sang Dang zkidkid@gmail.com wrote:

Hi All,
I am going to build a log central using Elasticsearch.
I need some advice from anyone who have built it already.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5af88543-5806-4021-83a5-41abc5b2bed6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5af88543-5806-4021-83a5-41abc5b2bed6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DFpaQo8togCSKFR5J6bsqMBw_8-39uNPPj_Q6H2ag%2Bow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Vineeth Mohan,
My log central will contain 2 type of log, one is for log to debug/monitor,
other is for stats.
I have 2 ways to achieve it:

#1 , I use only ES, it's ok to log for debug/monitor (using kibana).
To do stats, I will build some extra api (base on
filter/facet/agregration...)

#2, I use ES as external data storage, and write data to ES use Apache Hive
(https://github.com/elasticsearch/elasticsearch-hadoop#apache-hive)
this approach will help me alot in doing stats, but I don't know whether
it's good for logging other info ( to debug/monitor purpose).

I really appreciate your help :slight_smile:

Best Regards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20091177-e41d-45be-bd0a-c535d7c65871%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hello All,
I have selected #2 as my solution.
I write data to ES, and use kibana+ to realtime monitor.
For stats, I use Hive.

Each project, I will create a index, for each type of log I will put in a
ES Type,
ex: ProjectX>>log_debug
>>log_error
>>Stats_API
>>Stats_PageView
>>Stats_XYZ

I am wonder whether it's good ?
Should I separate by time for each type of project ?

Regards.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/35487688-4204-4f4d-aa2e-2a9b6a43aa82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Hello Sang ,

Can i know why you are using Hive.
I feel you can do the analysis in Elasticsearch itself.
Rest seems good to me.

Thanks
Vineeth

On Tue, Aug 26, 2014 at 8:03 AM, Sang Dang zkidkid@gmail.com wrote:

Hello All,
I have selected #2 as my solution.
I write data to ES, and use kibana+ to realtime monitor.
For stats, I use Hive.

Each project, I will create a index, for each type of log I will put in a
ES Type,
ex: ProjectX>>log_debug
>>log_error
>>Stats_API
>>Stats_PageView
>>Stats_XYZ

I am wonder whether it's good ?
Should I separate by time for each type of project ?

Regards.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/35487688-4204-4f4d-aa2e-2a9b6a43aa82%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/35487688-4204-4f4d-aa2e-2a9b6a43aa82%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5mvMAK%3DkKqg%3DTyzb-J0Boo_CVPUnC_vY0j%2BhNn_rH8_%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.