Hi,
I got a weird problem that i would like to share.
I have one filebeat instance trying to harvest a particular file and its just doing nothing.
Here's my filebeat config -
# ============================== Filebeat inputs ===============================
filebeat.config.inputs:
enabled: true
path: /etc/filebeat/conf.d/*.conf
________________________________________________________________________________
In /etc/filebeat/conf.d/*.conf i have only one config file -
- type: log
paths:
- /opt/perforce/logfiles/clelievre/p4logs.log
scan_frequency: 10s
close_removed: true
fields_under_root: true
fields:
ubi_team: "GNS Production/Source Control"
p4_instance: clelievre4950
p4_region: ncsa
p4_type: standard
log_type: p4d.log
multiline.pattern: '^--- '
multiline.negate: false
multiline.match: after
When i start Filebeat, everything seems ok -
tail -f /var/log/filebeat/filebeat
2020-11-20T13:46:57.911-0500 INFO log/harvester.go:302 Harvester started for file: /opt/perforce/logfiles/clelievre/p4logs.log
2020-11-20T13:47:00.891-0500 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-11-20T13:47:01.891-0500 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://ne1-sc-logstash01:5044))
2020-11-20T13:47:01.891-0500 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-11-20T13:47:01.892-0500 INFO [publisher] pipeline/retry.go:223 done
2020-11-20T13:47:01.892-0500 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://ne1-sc-logstash02:5044))
2020-11-20T13:47:01.892-0500 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-11-20T13:47:01.892-0500 INFO [publisher] pipeline/retry.go:223 done
2020-11-20T13:47:01.904-0500 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://ne1-sc-logstash02:5044)) established
2020-11-20T13:47:01.904-0500 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://ne1-sc-logstash01:5044)) established
Then, nothing happens, nothing is sent to both of my logstash servers.
If i "lsof" on the file that is supposed to be harvested -
lsof p4logs.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 24384 root 13r REG 253,21 4015 537325792 p4logs.log
If i strace the process 24384 -
strace -p 24384
Process 24384 attached
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
fstat(13, {st_mode=S_IFREG|0644, st_size=4015, ...}) = 0
futex(0x64d08f8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x64d07f8, FUTEX_WAKE_PRIVATE, 1) = 1
newfstatat(AT_FDCWD, "/opt/perforce/logfiles/clelievre/p4logs.log", {st_mode=S_IFREG|0644, st_size=4015, ...}, 0) = 0
futex(0x64d08f8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x64d07f8, FUTEX_WAKE_PRIVATE, 1) = 1
read(13, "", 16384) = 0
futex(0x64d08f8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x64d07f8, FUTEX_WAKE_PRIVATE, 1) = 1
fstat(13, {st_mode=S_IFREG|0644, st_size=4015, ...}) = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=24384, si_uid=0} ---
rt_sigreturn() = 824633931248
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc00068c148, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc00068c148, FUTEX_WAKE_PRIVATE, 1) = 1
fstat(13, {st_mode=S_IFREG|0644, st_size=4015, ...}) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000d804c8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
write(2, "\0", 1) = 1
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000d804c8, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x64d17a8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
My concern here is more about this message - EAGAIN (Resource temporarily unavailable) , where im not quite sure what it means, maybe handlers are fighting each other ?
p4logs.log is a test file for this problem, no software or anything have any handle on it, i just asked Filebeat to crawl it and send it to logstash server.
Its just doing nothing.
Anyone would have a suggestion on what to look for ?
Thank you.