# Logs not back filled if logstash indexer pipeline is down **URL:** \ **Category:** Logstash\ **Created:** [February 6, 2016, 11:57am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118 "2016-02-06T11:57:53Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\ **Post date:** [February 6, 2016, 11:57am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/1 "2016-02-06T11:57:53Z")
Hi Everyone, I am not able to figure out why logs are not back filled when indexer logstash is down although logs are buffered in kafka . Regards Arvind ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 6, 2016, 1:28pm UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/2 "2016-02-06T13:28:27Z")
A one-sentence question like this is impossible to answer. Have you configured Logstash to fetch logs from Kafka? And that stops working after Logstash has been down for a while? ---
**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\ **Post date:** [February 6, 2016, 1:33pm UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/3 "2016-02-06T13:33:42Z")
Configs would help... ---
**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\ **Post date:** [February 9, 2016, 8:14am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/4 "2016-02-09T08:14:07Z")
Sorry for incomplete information .. My logging architecture is as follows filebeat -\> shipper ( beat as input and kafka as output) - \> kafka \<- indexer (kafka input + filter for access log parsing + elastic as output) \<- kibana Following are the configs --- For Shipper input { beats { host =\> "X.X.X.X" port =\> 5044 congestion\_threshold =\> 30 } } output { if [type] == "access\_log" { kafka { retry\_backoff\_ms =\> 30 linger\_ms =\> 5 topic\_id =\> "access\_log" bootstrap\_servers =\> "X.X.X.X:9092,X.X.X.X:9092,X.X.X.X:9092" acks =\> "0" } } } For Indexer input { kafka { zk\_connect =\> "X.X.X.X:2181,X.X.X.X:2181,X.X.X.X:2181" group\_id =\> "access\_log" topic\_id =\> "access\_log" reset\_beginning =\> true consumer\_threads =\> 5 consumer\_restart\_on\_error =\> true consumer\_restart\_sleep\_ms =\> 100 decorate\_events =\> true } } filter { grok { match =\> { "message" =\> "%{COMBINEDAPACHELOG}" } patterns\_dir =\> ["/opt/logstash/grok/apache\_access"] } date { match =\> ["log\_time" , "dd/MMM/yyyy:HH:mm:ss Z"] target =\> "@timestamp" locale =\> "en" } mutate { remove\_field =\> ["log\_time" , "auth" , "ident" , "beat" , "input\_type" , "source"] } mutate { gsub =\> [ "referrer", "^"", "", "referrer", ""$", "", "agent", "^"", "", "agent", ""$", "" ] } useragent { prefix =\> "agent\_" source =\> "agent" remove\_field =\> ["agent\_major", "agent\_patch", "agent\_build", "agent\_minor", "agent\_os\_major", "agent\_os\_minor"] } } output { elasticsearch { index =\> "access\_log-%{+YYYY.MM.dd}" hosts =\> ["X.X.X.X:9210"] template =\> "/opt/logstash/template/accesslog.json" template\_overwrite =\> true template\_name =\> "access\_log-\*" workers =\> 2 } } f ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 9, 2016, 8:25am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/5 "2016-02-09T08:25:23Z")
Okay, and what's the problem? ---
**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\ **Post date:** [February 9, 2016, 9:24am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/6 "2016-02-09T09:24:15Z")
Problem is , Sometime indexer pipeline freezes , in this case i have to restart indexer , So from the time indexer was freeze till restart there are no logs , but indexer should back fill logs from kafka ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 9, 2016, 9:35am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/7 "2016-02-09T09:35:54Z")
Is there anything in the Logstash logs when this happens? ---
**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\ **Post date:** [February 9, 2016, 10:24am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/8 "2016-02-09T10:24:43Z")
There was nothing in logstash logs , It seems logstash is taking the last offset from zookeeper for the logs due to which old logs are not being fetched . not very sure about this . ---
**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\ **Post date:** [February 11, 2016, 12:31am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/9 "2016-02-11T00:31:29Z")
You want to remove reset\_beginning =\> true. That makes it delete your offsets every time Logstash restarts. ---
**Author:** ![astro](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@astro](https://discuss.elastic.co/u/astro)\ **Post date:** [February 11, 2016, 1:21pm UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/10 "2016-02-11T13:21:17Z")
Thanks Joe\_Lawson removing reset\_beginning served my requirement . 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:12am UTC](https://discuss.elastic.co/t/logs-not-back-filled-if-logstash-indexer-pipeline-is-down/41118/11 "2017-07-06T05:12:03Z")