Filebeat unable to ship non modified files to logstash

Elastic Stack Beats
1

Hello,

I configured filebeat.yml to send data to logstash, I just specified where my data are stored and the ip address of my logstash.

I have a python script to generate data, if I don't change any line in my data, filebeat don't send them to logstash. And after a modification filebeat succeed to send data to logstash.

I would like send these files to logstash without being constrained to modify them by hand each time.

Thank you for your help.

2

Please share the debug logs of Filebeat (output of ./filebeat -e -d "*").

3 
2019-08-06T14:11:00.693+0200    DEBUG   [acker] beater/acker.go:64      stateful ack {"count": 1}
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:404   Registry file updated. 172 states written.
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:356   Processing 1 events
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:326   Registrar state updates processed. Count: 1
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:346   Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.713+0200    DEBUG   [registrar]     registrar/registrar.go:411   Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.713+0200    DEBUG   [acker] beater/acker.go:64      stateful ack {"count": 1}
2019-08-06T14:11:00.713+0200    DEBUG   [publisher]     pipeline/client.go:200  Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {698022-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/54563ec9-d582-4641-8b14-6edafae9c65e.json 0 2019-08-06 14:08:59.312651456 +0200 CEST -1ns log map[] 698022-65025} false}
2019-08-06T14:11:00.713+0200    DEBUG   [input] file/states.go:68       New state added for /home/test/Documents/PRODUCERS/Data/HC/616fc149-9cae-4932-9691-e0b7ec2c521c.json
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:404   Registry file updated. 172 states written.
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:356   Processing 1 events
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:326   Registrar state updates processed. Count: 1
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:346   Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.733+0200    DEBUG   [registrar]     registrar/registrar.go:411   Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.733+0200    DEBUG   [publisher]     pipeline/client.go:200  Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {697952-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/616fc149-9cae-4932-9691-e0b7ec2c521c.json 0 2019-08-06 14:08:59.365041842 +0200 CEST -1ns log map[] 697952-65025} false}
2019-08-06T14:11:00.733+0200    DEBUG   [input] file/states.go:68       New state added for /home/test/Documents/PRODUCERS/Data/HC/6aba6cf8-377c-422a-8ca7-91e2b9995d87.json
2019-08-06T14:11:00.733+0200    DEBUG   [acker] beater/acker.go:64      stateful ack {"count": 1}
2019-08-06T14:11:00.757+0200    DEBUG   [registrar]     registrar/registrar.go:404   Registry file updated. 172 states written.
2019-08-06T14:11:00.757+0200    DEBUG   [registrar]     registrar/registrar.go:356   Processing 1 events
2019-08-06T14:11:00.758+0200    DEBUG   [registrar]     registrar/registrar.go:326   Registrar state updates processed. Count: 1
2019-08-06T14:11:00.758+0200    DEBUG   [registrar]     registrar/registrar.go:346   Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.758+0200    DEBUG   [registrar]     registrar/registrar.go:411   Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.758+0200    DEBUG   [publisher]     pipeline/client.go:200  Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {697970-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/6aba6cf8-377c-422a-8ca7-91e2b9995d87.json 0 2019-08-06 14:08:59.387379355 +0200 CEST -1ns log map[] 697970-65025} false}
2019-08-06T14:11:00.758+0200    DEBUG   [input] file/states.go:68       New state added for /home/test/Documents/PRODUCERS/Data/HC/7c80e9f7-f1a0-4485-90db-65dc27eb0d65.json
4

hello, you want me to provide more log ?

5

Does your script add a newline to the end of log lines it generates? Filebeat needs a newline at the end of a line in order to detect a finished log line.

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.