Filbeats not forwarding logs to logstash with back off now error

Elastic Stack Beats
1

I'm facing a problem to forward logs from filebeats to logstash, my filbeats always throw a debug value backoff now and the following are logs of it, would some help me and also please help me to reprocess the logs from logstash to elasticsearch by resetting

LocallevelArticleAttributes/LocallevelArticleAttributes_MsgLogger.log, offset: 644
2018-06-27T13:50:40.543Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I2639_LocallevelArticleAttributes/LocallevelArticleAttributes_MsgLogger.log
2018-06-27T13:50:40.543Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
2018-06-27T13:50:40.572Z DEBUG [input] input/input.go:124 Run input
2018-06-27T13:50:40.572Z DEBUG [input] log/input.go:147 Start next scan
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_AppLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_AppLogger.log, offset: 8514
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_AppLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_MsgLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_MsgLogger.log, offset: 914
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_ArticlePrice/ArticlePrice_MsgLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
2018-06-27T13:50:40.573Z DEBUG [input] input/input.go:124 Run input
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:147 Start next scan
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log, offset: 1784
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log, offset: 196
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log
2018-06-27T13:50:40.573Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
2018-06-27T13:50:40.577Z DEBUG [input] input/input.go:124 Run input
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:147 Start next scan
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_AppLogger.log
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_AppLogger.log, offset: 18080
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_AppLogger.log
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_MsgLogger.log
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_MsgLogger.log, offset: 208
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:502 File didn't change: /usr/local/applogs/RetailIS_R4/RT_I1273_GloballevelArticleAttributes/GloballevelArticleAttributes_MsgLogger.log
2018-06-27T13:50:40.577Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 2, After: 2, Pending: 0
2018-06-27T13:50:50.543Z DEBUG [input] input/input.go:124 Run input
2018-06-27T13:50:50.543Z DEBUG [input] log/input.go:147 Start next scan
2018-06-27T13:50:50.543Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_I2639_LocallevelArticleAttributes/LocallevelArticleAttributes

2

Filebeat does not reread files it has already encountered and forwarded to outputs.
Do you want to reread all logs processed by Filebeat?

3

Yes, I want to reprocess the files and I deleted the filebeat registry in data folder but still not able to reprocess them

4

@kvch Please help me with the filebeats

5

Could you share the output of ./filebeat -e -d "*" after deleting the registry file?

6

@kvch, the filebeats had processed the logs and the following is the output of logs

0619\u003c/GLTAB\u003e\u003c/E1WBBAH\u003e\u003c/E1WBB01\u003e\u003c/IDOC\u003e\u003c/ZWBBDLD05_G01\u003e",
"input": {
"type": "log"
},
"prospector": {
"type": "log"
},
"fields": {
"log_type": "GA"
},
"beat": {
"hostname": "xxxxx.com",
"version": "6.3.0",
"name": "xxxx.com"
},
"host": {
"name": "xxxxx.com"
},
"source": "/opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log"
}
2018-06-28T18:18:09.618Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:09.992Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log; Backoff now.
2018-06-28T18:18:10.046Z DEBUG [logstash] logstash/async.go:142 9 events out of 9 events sent to logstash host 172.23.36.82:5044. Continue sending
2018-06-28T18:18:10.117Z DEBUG [memqueue] memqueue/ackloop.go:143 ackloop: receive ack [5: 0, 9]
2018-06-28T18:18:10.117Z DEBUG [memqueue] memqueue/eventloop.go:518 broker ACK events: count=3, start-seq=10, end-seq=12

2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/eventloop.go:518 broker ACK events: count=4, start-seq=12, end-seq=15

2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/eventloop.go:518 broker ACK events: count=2, start-seq=11, end-seq=12

2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/ackloop.go:111 ackloop: return ack to broker loop:9
2018-06-28T18:18:10.118Z DEBUG [memqueue] memqueue/ackloop.go:114 ackloop: done send ack
2018-06-28T18:18:10.118Z DEBUG [acker] beater/acker.go:47 stateful ack {"count": 9}
2018-06-28T18:18:10.118Z DEBUG [registrar] registrar/registrar.go:232 Processing 9 events
2018-06-28T18:18:10.118Z DEBUG [registrar] registrar/registrar.go:202 Registrar state updates processed. Count: 9
2018-06-28T18:18:10.118Z DEBUG [registrar] registrar/registrar.go:263 Write registry file: /opt/webmis/filebeat-6.3.0-linux-x86_64/data/registry
2018-06-28T18:18:10.128Z DEBUG [registrar] registrar/registrar.go:290 Registry file updated. 5 states written.
2018-06-28T18:18:10.204Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:10.618Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:10.768Z DEBUG [harvester] log/log.go:85 End of file reached: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_MsgLogger.log; Backoff now.
2018-06-28T18:18:10.768Z DEBUG [harvester] log/log.go:85 End of file reached: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log; Backoff now.
2018-06-28T18:18:11.992Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log; Backoff now.
2018-06-28T18:18:12.204Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:12.618Z DEBUG [harvester] log/log.go:85 End of file reached: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log; Backoff now.
2018-06-28T18:18:13.307Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:362 Check file for harvesting: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log, offset: 6968086
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:500 Harvester for file is still running: /opt/webmis/ELKTest/GloballevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.307Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-28T18:18:13.316Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:362 Check file for harvesting: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log, offset: 6055946
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:500 Harvester for file is still running: /opt/webmis/ELKTest/LocallevelArticleAttributes_AppLogger.log
2018-06-28T18:18:13.316Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-28T18:18:13.318Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:362 Check file for harvesting: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log, offset: 5744535
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:500 Harvester for file is still running: /opt/webmis/ELKTest/ArticlePrice_AppLogger.log
2018-06-28T18:18:13.318Z DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-28T18:18:13.324Z DEBUG [input] input/input.go:124 Run input
2018-06-28T18:18:13.324Z DEBUG [input] log/input.go:147 Start next scan
2018-06-28T18:18:13.324Z DEBUG [input] log/input.go:362 Check file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log
2018-06-28T18:18:13.324Z DEBUG [input] log/input.go:448 Update existing file for harvesting: /usr/local/applogs/RetailIS_R4/RT_M4193_PricingPBC/PricingPBC_AppLogger.log, offset: 1784

7

If I understood you correctly, you need to change the configuration of Filebeat to forward logs to Logstash instead of Elasticsearch, delete the registry file and send the logs again.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.