# Log management with Elasticsearch? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 15, 2010, 1:30pm UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220 "2010-08-15T13:30:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Mike\_Pilkington](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@Mike\_Pilkington](https://discuss.elastic.co/u/Mike_Pilkington)\ **Post date:** [August 15, 2010, 1:30pm UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220/1 "2010-08-15T13:30:22Z")
Hi, I'm wondering if anyone has used Elasticsearch to manage (provide search for) large log file collections? I'm looking for solutions to help with a semi-centralized log management project. The logs would be sent in syslog-style format from hundreds of servers/routers/firewalls and maintained on a few dedicated log servers. I looked into Splunk, a popular log management solution which has the ability to scale horizontally (add more servers for more storage and performance). I assume it uses some sort of NoSQL technology. Unfortunately, their solution is too expensive and after searching for an open-source equivalent and not finding it, I'm looking into possibly building a home-grown solution. I came across the following blog post series on log management with Hadoop: [http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/](http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/). In the comments of the 3rd blog post, Elasticsearch was mentioned as a possibility and so I'm wondering if anyone out there has applied Elasticsearch to log management? Thanks, Mike ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [August 15, 2010, 1:58pm UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220/2 "2010-08-15T13:58:50Z")
Hi Mike, I think solution for the use case you're describing is combination of Flume and Elasticsearch. Flume provides great infrastructure to aggregate logs (has a syslog receiver) and all the data can be indexed in Elasticsearch for querying later. All that is needed is an Elasticsearch Sink, the component that would interface with ES, in flume terminology, which should be quite straight forward. Flume is an open source project recently released by Cloudera folks, user guide can be found here: [archive.cloudera.com/cdh/3/flume/UserGuide.html](http://archive.cloudera.com/cdh/3/flume/UserGuide.html) Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Sun, Aug 15, 2010 at 9:30 AM, Mike Pilkington [mpilking@gmail.com](mailto:mpilking@gmail.com) wrote: > Hi, > > I'm wondering if anyone has used Elasticsearch to manage (provide > search for) large log file collections? > > I'm looking for solutions to help with a semi-centralized log > management project. The logs would be sent in syslog-style format > from hundreds of servers/routers/firewalls and maintained on a few > dedicated log servers. I looked into Splunk, a popular log management > solution which has the ability to scale horizontally (add more servers > for more storage and performance). I assume it uses some sort of > NoSQL technology. Unfortunately, their solution is too expensive and > after searching for an open-source equivalent and not finding it, I'm > looking into possibly building a home-grown solution. > > I came across the following blog post series on log management with > Hadoop: > [http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/](http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/). > In the comments of the 3rd blog post, Elasticsearch was mentioned as > a possibility and so I'm wondering if anyone out there has applied > Elasticsearch to log management? > > Thanks, > Mike ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [September 11, 2010, 11:32pm UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220/3 "2010-09-11T23:32:19Z")
This post was a few weeks ago, but I've started to put something together for this on github, if anyone would like to collaborate. Very basic structure now there, and works, still need to consider how a fire hose stream of logs will cope, and whether one can perform batch indexing to improve performance. It is very simple to do though by the looks of it. [http://github.com/tallpsmith/elasticflume](http://github.com/tallpsmith/elasticflume) cheers, Paul On Aug 15, 11:58 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote: > Hi Mike, > > I think solution for the use case you're describing is combination of Flume > and Elasticsearch. Flume provides great infrastructure to aggregate logs > (has a syslog receiver) and all the data can be indexed in Elasticsearch for > querying later. All that is needed is an Elasticsearch Sink, the component > that would interface with ES, in flume terminology, which should be quite > straight forward. > > Flume is an open source project recently released by Cloudera folks, user > guide can be found here: > [archive.cloudera.com/cdh/3/flume/UserGuide.html](http://archive.cloudera.com/cdh/3/flume/UserGuide.html) > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Sun, Aug 15, 2010 at 9:30 AM, Mike Pilkington [mpilk...@gmail.com](mailto:mpilk...@gmail.com) wrote: > > > Hi, > > > I'm wondering if anyone has used Elasticsearch to manage (provide > > search for) large log file collections? > > > I'm looking for solutions to help with a semi-centralized log > > management project. The logs would be sent in syslog-style format > > from hundreds of servers/routers/firewalls and maintained on a few > > dedicated log servers. I looked into Splunk, a popular log management > > solution which has the ability to scale horizontally (add more servers > > for more storage and performance). I assume it uses some sort of > > NoSQL technology. Unfortunately, their solution is too expensive and > > after searching for an open-source equivalent and not finding it, I'm > > looking into possibly building a home-grown solution. > > > I came across the following blog post series on log management with > > Hadoop: > > [http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/](http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/). > > In the comments of the 3rd blog post, Elasticsearch was mentioned as > > a possibility and so I'm wondering if anyone out there has applied > > Elasticsearch to log management? > > > Thanks, > > Mike ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 12, 2010, 8:30am UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220/4 "2010-09-12T08:30:34Z")
Looks great!. I will add it to the projects page. I think this: [http://github.com/tallpsmith/elasticflume/blob/master/src/main/java/org/elasticsearch/flume/ElasticSearchSink.javaspeaks](http://github.com/tallpsmith/elasticflume/blob/master/src/main/java/org/elasticsearch/flume/ElasticSearchSink.javaspeaks) volumes on the simplicity of both elasticsearch and flume! On Sun, Sep 12, 2010 at 1:32 AM, tallpsmith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > This post was a few weeks ago, but I've started to put something > together for this on github, if anyone would like to collaborate. > Very basic structure now there, and works, still need to consider how > a fire hose stream of logs will cope, and whether one can perform > batch indexing to improve performance. It is very simple to do though > by the looks of it. > > [http://github.com/tallpsmith/elasticflume](http://github.com/tallpsmith/elasticflume) > > cheers, > > Paul > > On Aug 15, 11:58 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) > wrote: > > > Hi Mike, > > > > I think solution for the use case you're describing is combination of > > Flume > > and Elasticsearch. Flume provides great infrastructure to aggregate logs > > (has a syslog receiver) and all the data can be indexed in Elasticsearch > > for > > querying later. All that is needed is an Elasticsearch Sink, the > > component > > that would interface with ES, in flume terminology, which should be quite > > straight forward. > > > > Flume is an open source project recently released by Cloudera folks, user > > guide can be found here: > > [archive.cloudera.com/cdh/3/flume/UserGuide.html](http://archive.cloudera.com/cdh/3/flume/UserGuide.html) > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Sun, Aug 15, 2010 at 9:30 AM, Mike Pilkington [mpilk...@gmail.com](mailto:mpilk...@gmail.com) > > wrote: > > > > > Hi, > > > > > I'm wondering if anyone has used Elasticsearch to manage (provide > > > search for) large log file collections? > > > > > I'm looking for solutions to help with a semi-centralized log > > > management project. The logs would be sent in syslog-style format > > > from hundreds of servers/routers/firewalls and maintained on a few > > > dedicated log servers. I looked into Splunk, a popular log management > > > solution which has the ability to scale horizontally (add more servers > > > for more storage and performance). I assume it uses some sort of > > > NoSQL technology. Unfortunately, their solution is too expensive and > > > after searching for an open-source equivalent and not finding it, I'm > > > looking into possibly building a home-grown solution. > > > > > I came across the following blog post series on log management with > > > Hadoop: > > > [http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/](http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/) > > > . > > > In the comments of the 3rd blog post, Elasticsearch was mentioned as > > > a possibility and so I'm wondering if anyone out there has applied > > > Elasticsearch to log management? > > > > > Thanks, > > > Mike ---
**Author:** ![Ted\_Karmel](https://avatars.discourse-cdn.com/v4/letter/t/e9a140/32.png) [@Ted\_Karmel](https://discuss.elastic.co/u/Ted_Karmel)\ **Post date:** [September 12, 2010, 12:45pm UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220/5 "2010-09-12T12:45:58Z")
+1 for Shay's comment... Paul, I was considering this very option and just saw your email and git on the subject. On Sun, Sep 12, 2010 at 9:30 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Looks great!. I will add it to the projects page. I think this: > [http://github.com/tallpsmith/elasticflume/blob/master/src/main/java/org/elasticsearch/flume/ElasticSearchSink.java](http://github.com/tallpsmith/elasticflume/blob/master/src/main/java/org/elasticsearch/flume/ElasticSearchSink.java) > speaks volumes on the simplicity of both elasticsearch and flume! > > On Sun, Sep 12, 2010 at 1:32 AM, tallpsmith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > > > This post was a few weeks ago, but I've started to put something > > together for this on github, if anyone would like to collaborate. > > Very basic structure now there, and works, still need to consider how > > a fire hose stream of logs will cope, and whether one can perform > > batch indexing to improve performance. It is very simple to do though > > by the looks of it. > > > > [http://github.com/tallpsmith/elasticflume](http://github.com/tallpsmith/elasticflume) > > > > cheers, > > > > Paul > > > > On Aug 15, 11:58 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) > > wrote: > > > > > Hi Mike, > > > > > > I think solution for the use case you're describing is combination of > > > Flume > > > and Elasticsearch. Flume provides great infrastructure to aggregate logs > > > (has a syslog receiver) and all the data can be indexed in Elasticsearch > > > for > > > querying later. All that is needed is an Elasticsearch Sink, the > > > component > > > that would interface with ES, in flume terminology, which should be > > > quite > > > straight forward. > > > > > > Flume is an open source project recently released by Cloudera folks, > > > user > > > guide can be found here: > > > [archive.cloudera.com/cdh/3/flume/UserGuide.html](http://archive.cloudera.com/cdh/3/flume/UserGuide.html) > > > > > > Regards, > > > Berkay Mollamustafaoglu > > > mberkay on yahoo, google and skype > > > > > > On Sun, Aug 15, 2010 at 9:30 AM, Mike Pilkington [mpilk...@gmail.com](mailto:mpilk...@gmail.com) > > > wrote: > > > > > > > Hi, > > > > > > > I'm wondering if anyone has used Elasticsearch to manage (provide > > > > search for) large log file collections? > > > > > > > I'm looking for solutions to help with a semi-centralized log > > > > management project. The logs would be sent in syslog-style format > > > > from hundreds of servers/routers/firewalls and maintained on a few > > > > dedicated log servers. I looked into Splunk, a popular log management > > > > solution which has the ability to scale horizontally (add more servers > > > > for more storage and performance). I assume it uses some sort of > > > > NoSQL technology. Unfortunately, their solution is too expensive and > > > > after searching for an open-source equivalent and not finding it, I'm > > > > looking into possibly building a home-grown solution. > > > > > > > I came across the following blog post series on log management with > > > > Hadoop: > > > > > > > > [http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/](http://blog.mgm-tp.com/series/scalable-log-data-management-with-hadoop/). > > > > > In the comments of the 3rd blog post, Elasticsearch was mentioned as > > > > > a possibility and so I'm wondering if anyone out there has applied > > > > > Elasticsearch to log management? > > > > > > > Thanks, > > > > Mike ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:19am UTC](https://discuss.elastic.co/t/log-management-with-elasticsearch/3220/6 "2017-07-06T04:19:28Z")