# Flume-NG ElasticSearch Sink Backing up @ Midnight **URL:** \ **Category:** Elasticsearch\ **Created:** [April 10, 2014, 4:03pm UTC](https://discuss.elastic.co/t/flume-ng-elasticsearch-sink-backing-up-midnight/16930 "2014-04-10T16:03:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Matt\_Wise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_wise/32/701_2.png) [@Matt\_Wise](https://discuss.elastic.co/u/Matt_Wise)\ **Post date:** [April 10, 2014, 4:03pm UTC](https://discuss.elastic.co/t/flume-ng-elasticsearch-sink-backing-up-midnight/16930/1 "2014-04-10T16:03:25Z")
We use Flume 1.4 to pass logs into HDFS as well as Elasticsearch for storage. The pipeline looks roughly like this: Client to Server Flow... (local\_app -\> local\_host\_flume\_agent) ---- AVRO/SSL ----\> (remote\_flume\_agent)... Agent Server Flow ... (inbound avro -\> FC1 -\> Elasticsearch) (inbound avro -\> FC2 -\> S3/HDFS) In the last week we've made a few changes and now we're seeing a bit of a problem. We'e seen 3 different occurrences of a single flume agent server node beginning to back up its FC1 channel indefinitely until we log in and restart Flume entirely. The data just stops flowing -- we can't find any errors in the logs on either the ES or Flume side. A simple restart of Flume fixes it. Our sink config looks like this: > agent.sinks.elasticsearch.type = > org.apache.flume.sink.elasticsearch.ElasticSearchSink > agent.sinks.elasticsearch.hostNames = xxx:9300 > agent.sinks.elasticsearch.indexName = flume > agent.sinks.elasticsearch.clusterName = > flume-elasticsearch-production-useast1 > agent.sinks.elasticsearch.batchSize = 1000 > agent.sinks.elasticsearch.ttl = 30 > agent.sinks.elasticsearch.serializer = > org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer > agent.sinks.elasticsearch.channel = fc-unstructured-es This ONLY happens at Midnight, and only happens on one flume server. I'm wondering whether it has to do with the time it takes our ES nodes to create a new index ... and the first flume agent that triggers "index creation" could be getting blocked or stuck? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7892490-d2f6-442f-ae25-18b59021e7e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7892490-d2f6-442f-ae25-18b59021e7e4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Wise](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt_wise/32/701_2.png) [@Matt\_Wise](https://discuss.elastic.co/u/Matt_Wise)\ **Post date:** [April 10, 2014, 4:12pm UTC](https://discuss.elastic.co/t/flume-ng-elasticsearch-sink-backing-up-midnight/16930/2 "2014-04-10T16:12:07Z")
One additional thing.. we have two ES sinks actually pointing to the same cluster. The config looks more like this actually: (inbound avro -\> FC1 -\> Elasticsearch) (inbound avro -\> FC2 -\> S3/HDFS) (inbound avro\_2 -\> FC3 -\> Elasticsearch) (inbound avro\_2 -\> FC4 -\> S3/HDFS) On Thursday, April 10, 2014 9:03:25 AM UTC-7, Matt wrote: > We use Flume 1.4 to pass logs into HDFS as well as Elasticsearch for > storage. The pipeline looks roughly like this: > > Client to Server Flow... > (local\_app -\> local\_host\_flume\_agent) ---- AVRO/SSL ----\> > (remote\_flume\_agent)... > > Agent Server Flow ... > (inbound avro -\> FC1 -\> Elasticsearch) > (inbound avro -\> FC2 -\> S3/HDFS) > > In the last week we've made a few changes and now we're seeing a bit of a > problem. We'e seen 3 different occurrences of a single flume agent server > node beginning to back up its FC1 channel indefinitely until we log in and > restart Flume entirely. The data just stops flowing -- we can't find any > errors in the logs on either the ES or Flume side. A simple restart of > Flume fixes it. > > Our sink config looks like this: > > > agent.sinks.elasticsearch.type = > > org.apache.flume.sink.elasticsearch.ElasticSearchSink > > agent.sinks.elasticsearch.hostNames = xxx:9300 > > agent.sinks.elasticsearch.indexName = flume > > agent.sinks.elasticsearch.clusterName = > > flume-elasticsearch-production-useast1 > > agent.sinks.elasticsearch.batchSize = 1000 > > agent.sinks.elasticsearch.ttl = 30 > > agent.sinks.elasticsearch.serializer = > > org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer > > agent.sinks.elasticsearch.channel = fc-unstructured-es > > This ONLY happens at Midnight, and only happens on one flume server. I'm > wondering whether it has to do with the time it takes our ES nodes to > create a new index ... and the first flume agent that triggers "index > creation" could be getting blocked or stuck? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3056add8-40e8-4156-b8d6-834f34baf8c0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3056add8-40e8-4156-b8d6-834f34baf8c0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [April 10, 2014, 6:34pm UTC](https://discuss.elastic.co/t/flume-ng-elasticsearch-sink-backing-up-midnight/16930/3 "2014-04-10T18:34:07Z")
Matt, I don't know if this helps, but we are seeing similar issues with Flume using log4j2 (not log4j v1 as used by ES). For tomcat-hosted servlets, flume failover works fine. But for non-tomcat applications (such as looping batch-mode applications and Netty-based servers with static main entry points), we have found that when one of their flume loggers fails, there is no failover. We don't have a solution. But a workaround is that the non-tomcat applications only configure their log4j to write to one flume agent. If that fails, events are queued until the agent comes back up. No failover, but no data loss either. Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e51a0926-7742-4cd7-877d-1c1d12ed2a1d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e51a0926-7742-4cd7-877d-1c1d12ed2a1d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:36am UTC](https://discuss.elastic.co/t/flume-ng-elasticsearch-sink-backing-up-midnight/16930/4 "2017-07-06T01:36:48Z")