Filebeat unable to ship non modified files to logstash

Hello,

I configured filebeat.yml to send data to logstash, I just specified where my data are stored and the ip address of my logstash.

I have a python script to generate data, if I don't change any line in my data, filebeat don't send them to logstash. And after a modification filebeat succeed to send data to logstash.

I would like send these files to logstash without being constrained to modify them by hand each time.

Thank you for your help.

Please share the debug logs of Filebeat (output of ./filebeat -e -d "*").

2019-08-06T14:11:00.693+0200    DEBUG   [acker] beater/acker.go:64      stateful ack {"count": 1}
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:404   Registry file updated. 172 states written.
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:356   Processing 1 events
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:326   Registrar state updates processed. Count: 1
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:346   Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:411   Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.713+0200    DEBUG   [acker] beater/acker.go:64      stateful ack {"count": 1}
2019-08-06T14:11:00.713+0200    DEBUG   [publisher]     pipeline/client.go:200  Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {698022-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/54563ec9-d582-4641-8b14-6edafae9c65e.json 0 2019-08-06 14:08:59.312651456 +0200 CEST -1ns log map[] 698022-65025} false}
2019-08-06T14:11:00.713+0200    DEBUG   [input] file/states.go:68       New state added for /home/test/Documents/PRODUCERS/Data/HC/616fc149-9cae-4932-9691-e0b7ec2c521c.json
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:404   Registry file updated. 172 states written.
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:356   Processing 1 events
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:326   Registrar state updates processed. Count: 1
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:346   Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:411   Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.733+0200    DEBUG   [publisher]     pipeline/client.go:200  Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {697952-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/616fc149-9cae-4932-9691-e0b7ec2c521c.json 0 2019-08-06 14:08:59.365041842 +0200 CEST -1ns log map[] 697952-65025} false}
2019-08-06T14:11:00.733+0200    DEBUG   [input] file/states.go:68       New state added for /home/test/Documents/PRODUCERS/Data/HC/6aba6cf8-377c-422a-8ca7-91e2b9995d87.json
2019-08-06T14:11:00.733+0200    DEBUG   [acker] beater/acker.go:64      stateful ack {"count": 1}
2019-08-06T14:11:00.757+0200    DEBUG   [registrar]     registrar/registrar.go:404   Registry file updated. 172 states written.
2019-08-06T14:11:00.757+0200    DEBUG   [registrar]     registrar/registrar.go:356   Processing 1 events
2019-08-06T14:11:00.758+0200    DEBUG   [registrar]     registrar/registrar.go:326   Registrar state updates processed. Count: 1
2019-08-06T14:11:00.758+0200    DEBUG   [registrar]     registrar/registrar.go:346   Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.758+0200    DEBUG   [registrar]     registrar/registrar.go:411   Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.758+0200    DEBUG   [publisher]     pipeline/client.go:200  Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {697970-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/6aba6cf8-377c-422a-8ca7-91e2b9995d87.json 0 2019-08-06 14:08:59.387379355 +0200 CEST -1ns log map[] 697970-65025} false}
2019-08-06T14:11:00.758+0200    DEBUG   [input] file/states.go:68       New state added for /home/test/Documents/PRODUCERS/Data/HC/7c80e9f7-f1a0-4485-90db-65dc27eb0d65.json

hello, you want me to provide more log ?

Does your script add a newline to the end of log lines it generates? Filebeat needs a newline at the end of a line in order to detect a finished log line.

1 Like