Hello,
I configured filebeat.yml to send data to logstash, I just specified where my data are stored and the ip address of my logstash.
I have a python script to generate data, if I don't change any line in my data, filebeat don't send them to logstash. And after a modification filebeat succeed to send data to logstash.
I would like send these files to logstash without being constrained to modify them by hand each time.
Thank you for your help.
kvch
(Noémi Ványi)
August 6, 2019, 11:59am
2
Please share the debug logs of Filebeat (output of ./filebeat -e -d "*"
).
saisimo02
(Mohamed)
August 6, 2019, 12:16pm
3
kvch:
./filebeat -e -d "*"
2019-08-06T14:11:00.693+0200 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2019-08-06T14:11:00.713+0200 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 172 states written.
2019-08-06T14:11:00.713+0200 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2019-08-06T14:11:00.713+0200 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2019-08-06T14:11:00.713+0200 DEBUG [registrar] registrar/registrar.go:346 Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.713+0200 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.713+0200 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2019-08-06T14:11:00.713+0200 DEBUG [publisher] pipeline/client.go:200 Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {698022-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/54563ec9-d582-4641-8b14-6edafae9c65e.json 0 2019-08-06 14:08:59.312651456 +0200 CEST -1ns log map[] 698022-65025} false}
2019-08-06T14:11:00.713+0200 DEBUG [input] file/states.go:68 New state added for /home/test/Documents/PRODUCERS/Data/HC/616fc149-9cae-4932-9691-e0b7ec2c521c.json
2019-08-06T14:11:00.733+0200 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 172 states written.
2019-08-06T14:11:00.733+0200 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2019-08-06T14:11:00.733+0200 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2019-08-06T14:11:00.733+0200 DEBUG [registrar] registrar/registrar.go:346 Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.733+0200 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.733+0200 DEBUG [publisher] pipeline/client.go:200 Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {697952-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/616fc149-9cae-4932-9691-e0b7ec2c521c.json 0 2019-08-06 14:08:59.365041842 +0200 CEST -1ns log map[] 697952-65025} false}
2019-08-06T14:11:00.733+0200 DEBUG [input] file/states.go:68 New state added for /home/test/Documents/PRODUCERS/Data/HC/6aba6cf8-377c-422a-8ca7-91e2b9995d87.json
2019-08-06T14:11:00.733+0200 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2019-08-06T14:11:00.757+0200 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 172 states written.
2019-08-06T14:11:00.757+0200 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2019-08-06T14:11:00.758+0200 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2019-08-06T14:11:00.758+0200 DEBUG [registrar] registrar/registrar.go:346 Registrar states cleaned up. Before: 172, After: 172, Pending: 0
2019-08-06T14:11:00.758+0200 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (172)
2019-08-06T14:11:00.758+0200 DEBUG [publisher] pipeline/client.go:200 Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {697970-65025 true <nil> /home/test/Documents/PRODUCERS/Data/HC/6aba6cf8-377c-422a-8ca7-91e2b9995d87.json 0 2019-08-06 14:08:59.387379355 +0200 CEST -1ns log map[] 697970-65025} false}
2019-08-06T14:11:00.758+0200 DEBUG [input] file/states.go:68 New state added for /home/test/Documents/PRODUCERS/Data/HC/7c80e9f7-f1a0-4485-90db-65dc27eb0d65.json
hello, you want me to provide more log ?
kvch
(Noémi Ványi)
August 9, 2019, 3:30pm
5
Does your script add a newline to the end of log lines it generates? Filebeat needs a newline at the end of a line in order to detect a finished log line.
2 Likes
system
(system)
Closed
September 6, 2019, 3:30pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.