Filbeats not forwarding logs to logstash with back off now error

01n12 · June 27, 2018, 1:56pm

I'm facing a problem to forward logs from filebeats to logstash, my filbeats always throw a debug value backoff now and the following are logs of it, would some help me and also please help me to reprocess the logs from logstash to elasticsearch by resetting

LocallevelArticleAtt 2018-06-27T13:50:40.543Z 2018-06-27T13:50:40.543Z 2018-06-27T13:50:40.572Z 2018-06-27T13:50:40.572Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.573Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:40.577Z 2018-06-27T13:50:50.543Z 2018-06-27T13:50:50.543Z 2018-06-27T13:50:50.543Z ributes/LocallevelArticleAttributes_MsgLogger.log, offset: 644
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I2639_LocallevelArticleAttributes/LocallevelArticleAttributes_MsgLogger.log
DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
DEBUG [input] input/input.go:124 Run input
DEBUG [input] log/input.go:147 Start next scan
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_AppLogger.log
DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_AppLogger.log, offset: 8514
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_AppLogger.log
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_MsgLogger.log
DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_MsgLogger.log, offset: 914
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_MsgLogger.log
DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
DEBUG [input] input/input.go:124 Run input
DEBUG [input] log/input.go:147 Start next scan
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log
DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log, offset: 1784
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log
DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log, offset: 196
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log
DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
DEBUG [input] input/input.go:124 Run input
DEBUG [input] log/input.go:147 Start next scan
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_AppLogger.log
DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_AppLogger.log, offset: 18080
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_AppLogger.log
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_MsgLogger.log
DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_MsgLogger.log, offset: 208
DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_MsgLogger.log
DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
DEBUG [input] input/input.go:124 Run input
DEBUG [input] log/input.go:147 Start next scan
DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I2639_LocallevelArticleAttributes/LocallevelArticleAttributes

kvch · June 27, 2018, 3:17pm

Filebeat does not reread files it has already encountered and forwarded to outputs.
Do you want to reread all logs processed by Filebeat?

01n12 · June 27, 2018, 9:56pm

Yes, I want to reprocess the files and I deleted the filebeat registry in data folder but still not able to reprocess them

warrior · June 28, 2018, 4:54am

@kvch Please help me with the filebeats

kvch · June 28, 2018, 6:22am

Could you share the output of ./filebeat -e -d "*" after deleting the registry file?

warrior · June 28, 2018, 6:24pm

@kvch, the filebeats had processed the logs and the following is the output of logs

0619\u003c/GLTAB\u003e\u003c/E1WBBAH\u003e\u003c/E1WBB01\u003e\u003c/IDOC\u003e\u003c/ZWBBDLD05_G01\u003e",
"input": {
"type": "log"
},
"prospector": {
"type": "log"
},
"fields": {
"log_type": "GA"
},
"beat": {
"hostname": "xxxxx.com",
"version": "6.3.0",
"name": "xxxx.com"
},
"host": {
"name": "xxxxx.com"
},
"source": "/opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log"
}
2018-06-28T18:18:09.618Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:09.992Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log; Backoff now.
2018-06-28T18:18:10.046Z DEBUG [logstash] logstash/async.go:142 9 events out of 9 events sent to logstash host 172.23.36.82:5044. Continue sending
2018-06-28T18:18:10.117Z DEBUG [memqueue] memqueue/ackloop.go:143 ackloop: receive ack [5: 0, 9]
2018-06-28T18:18:10.117Z DEBUG [memqueue] memqueue/eventloop.go:518 broker ACK events: count=3, start-seq=10, end-seq=12

2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/eventloop.go:518 broker ACK events: count=4, start-seq=12, end-seq=15

2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/eventloop.go:518 broker ACK events: count=2, start-seq=11, end-seq=12

2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/ackloop.go:111 ackloop: return ack to broker loop:9
2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/ackloop.go:114 ackloop: done send ack
2018-06-28T18:18:10.118Z DEBUG [acker] beater/acker.go:47 stateful ack {"count": 9}
2018-06-28T18:18:10.118Z DEBUG [registrar] registrar/registrar.go:232 Processing 9 events
2018-06-28T18:18:10.118Z DEBUG [registrar] registrar/registrar.go:202 Registrar state updates processed. Count: 9
2018-06-28T18:18:10.118Z DEBUG [registrar] registrar/registrar.go:263 Write registry file: /opt/webmis/filebeat-6.3.0-linux-x86_64/data/registry
2018-06-28T18:18:10.128Z DEBUG [registrar] registrar/registrar.go:290 Registry file updated. 5 states written.
2018-06-28T18:18:10.204Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:10.618Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:10.768Z DEBUG [harvester] log/log.go:85 End of file reached: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log; Backoff now.
2018-06-28T18:18:10.768Z DEBUG [harvester] log/log.go:85 End of file reached: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log; Backoff now.
2018-06-28T18:18:11.992Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log; Backoff now.
2018-06-28T18:18:12.204Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:12.618Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:13.307Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:362 Check file for harvesting: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log, offset: 6968086
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:500 Harvester for file is still running: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-28T18:18:13.316Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:362 Check file for harvesting: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log, offset: 6055946
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:500 Harvester for file is still running: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-28T18:18:13.318Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:362 Check file for harvesting: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log, offset: 5744535
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:500 Harvester for file is still running: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-28T18:18:13.324Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.324Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.324Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log
2018-06-28T18:18:13.324Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log, offset: 1784

kvch · July 6, 2018, 12:58pm

If I understood you correctly, you need to change the configuration of Filebeat to forward logs to Logstash instead of Elasticsearch, delete the registry file and send the logs again.

system · August 3, 2018, 12:58pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.