Filebeat error to publish event because EOF

Tr_ng_Trang · February 14, 2017, 9:05am

hi sir,
I have a problem about filebeat, the problem need to solve soon( its important), so i sorry because need to send message direct to ask you. :(.
I have a folder, have some log file, its synchronous with folder tftp ( with other tftp server) so file log have the same name like: abc.log will be replaced by another file abc.log (i can't change name file because some reason, but i can't talk more) so its can summaries that, file log filebeat harverst will be the same name, but can same content or other. filebeat log in /var/log/filebeat show error to publish file in this file because error EOF. do you have any idea about this error.
Thanks you so much guys.

tudor · February 14, 2017, 11:02am

Can you post the actual log from Filebeat and the configuration file that you are using? Hitting EOF is usually not an error, but an expected event in Filebeat.

Tr_ng_Trang · February 15, 2017, 2:42am

sorry but i forget to save my log before i change logging level of file beat to Debug, so i just have log debug like:
2017-02-15T04:40:55+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActUserSpace_A0_65_18_08_78_2D_0.log
2017-02-15T04:40:55+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActUserSpace_A0_65_18_08_78_2D_0.log, offset: 868
2017-02-15T04:40:55+07:00 DBG File didn't change: /etc/tftproot/CscPSIActUserSpace_A0_65_18_08_78_2D_0.log
2017-02-15T04:40:55+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActUserSpace_A0_65_18_43_08_6E_0.log
2017-02-15T04:40:55+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActUserSpace_A0_65_18_43_08_6E_0.log, offset: 430
2017-02-15T04:40:55+07:00 DBG File didn't change: /etc/tftproot/CscPSIActUserSpace_A0_65_18_43_08_6E_0.log
2017-02-15T04:40:55+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActUserSpace_A0_65_18_66_91_64_0.log
2017-02-15T04:40:55+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActUserSpace_A0_65_18_66_91_64_0.log, offset: 331
2017-02-15T04:40:55+07:00 DBG File didn't change: /etc/tftproot/CscPSIActUserSpace_A0_65_18_66_91_64_0.log
2017-02-15T04:40:55+07:00 DBG Prospector states cleaned up. Before: 15, After: 15
2017-02-15T04:40:57+07:00 DBG End of file reached: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_85_0B_E3_0.log; Backoff now.
2017-02-15T04:40:57+07:00 DBG Run prospector
2017-02-15T04:40:57+07:00 DBG Start next scan
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_3.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_3.log, offset: 459
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_3.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_testlog.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_testlog.log, offset: 1617
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_testlog.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_0.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_0.log, offset: 104491
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_0.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_1.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_1.log, offset: 104618
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_1.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_2.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_2.log, offset: 104467
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_2.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_3.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_3.log, offset: 459
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_3.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_74_BA_0.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_74_BA_0.log, offset: 3738
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_74_BA_0.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_74_BA_6.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_74_BA_6.log, offset: 551
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_74_BA_6.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_0.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_0.log, offset: 104597
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_66_91_64_0.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_85_0B_E3_0.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_85_0B_E3_0.log, offset: 7571
2017-02-15T04:40:57+07:00 DBG Harvester for file is still running: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_85_0B_E3_0.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_08_78_2D_0.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_08_78_2D_0.log, offset: 459
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_08_78_2D_0.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_1.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_1.log, offset: 104585
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_1.log
2017-02-15T04:40:57+07:00 DBG Check file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_2.log
2017-02-15T04:40:57+07:00 DBG Update existing file for harvesting: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_2.log, offset: 104689
2017-02-15T04:40:57+07:00 DBG File didn't change: /etc/tftproot/CscPSIActKernelSpace_A0_65_18_43_08_6E_2.log
2017-02-15T04:40:57+07:00 DBG Prospector states cleaned up. Before: 15, After: 15
2017-02-15T04:41:00+07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-15T04:41:05+07:00 DBG Flushing spooler because of timeout. Events flushed: 0
2017-02-15T04:41:05+07:00 DBG Run prospector
2017-02-15T04:41:05+07:00 DBG Start next scan
2017-02-15T04:41:05+07:00 DBG Check file for harvesting:
it really have new file log in dir filebeat read log, but its can't publish event.

tudor · February 15, 2017, 11:23am

Can you also post your configuration file, please?

Tr_ng_Trang · February 16, 2017, 7:27am

sry for my late. this is my configuration in filebeat. i think its normal.
filebeat.prospectors:

paths:
- "/var/log/CscPSIActUserSpace_A0_65_18_54_B0_FC*"
  fields: {log_type: user_log}
paths:
- "/var/log/CscPSIActKernelSpace_A0_65_18_54_B0_FC*"
  fields: {log_type: sys_log}
  registry_file: /var/lib/filebeat/registry

ruflin · February 16, 2017, 9:57pm

Can you share the full log file and format it with 3 ticks around it so the indentation is correct?

Tr_ng_Trang · February 17, 2017, 3:08am

i check my configure, its right, filebeat already running correct. But now, i dont need to solve this problem, i have another problem with filebeat, it can't read log have multi line, but i have configured to multi line like:

The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

multiline.pattern: ^[LOG]

Defines if the pattern set under pattern should be negated or not. Default is false.

multiline.negate: true

Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

that was (not) matched before or after or as long as a pattern is not matched based on negate.

Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

multiline.match: after
and my log have format like:
[LOG]afafdsf
adf
dsf
sf
ffafad

but if i have just 1 file log, it working good, but if my filebeat configure to receive 2 source file log or more, its not working right.

ruflin · February 17, 2017, 7:24am

Can you open a new topic as this will not match the title anymore? Please make sure if you post configs to put it in as code for the formatting.

Tr_ng_Trang · February 20, 2017, 2:59am

thanks for your feedback, i will do it next time.

system · March 20, 2017, 2:59am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.