Filebeat: Error while retrieving FD information: error getting number of open FD: key not found

Elastic Stack Beats
1

I also got the same error as this post.
ERROR instance/metrics_file_descriptors.go:39 Error while retrieving FD information: error getting number of open FD: key not found

I wonder what is the cause of this error?
And, I also want to know how could I solve it?

Filebeat Version: filebeat version 6.5.4 (amd64), libbeat 6.5.4
OS Version: CentOS release 6.3 (Final)

2

On linux filebeat tries to get the number of file descriptors from the /proc/<pid>/fd file. The error indicates that file descriptor usage is not available in or the file can not be read by the process.

3

@steffens

I have confirmed the permission /proc/<pid>/fd is as below.
dr-x------ 2 root root 0 Apr 26 08:49 fd

I am executing Filebeat by using custom user rather than root. I don't think the process has permission to read the file.

In order to solve it, what should I do?

4

can you run:

$ psof filebeat

and 

$ ls -la /proc/$(psof filebeat)/

I'd assume filebeat should be able to read it's own state. Actually it would be better if filebeat did read /proc/self.

5

@steffens

Sorry for the late response.
I cannot find the command of psof in my server nor Google.
Do you mean lsof instead?

An error is occurred when I executed lsof

lsof: WARNING: can't stat() devtmpfs file system /var/unbound/dev/log
Output information may be incomplete.
lsof: WARNING: can't stat() devtmpfs file system /var/unbound/dev/random
Output information may be incomplete.
lsof: status error on filebeat: No such file or directory

For the result of ls -la /proc/<pid> is as below

ls: cannot read symbolic link /proc/13614/cwd: Permission denied
ls: cannot read symbolic link /proc/13614/root: Permission denied
ls: cannot read symbolic link /proc/13614/exe: Permission denied
total 0
dr-xr-xr-x 7 customuser root 0 Apr 26 08:49 .
dr-xr-xr-x 173 root root 0 Dec 7 2017 ..
dr-xr-xr-x 2 customuser root 0 Apr 26 08:49 attr
-rw-r--r-- 1 root root 0 Apr 26 08:49 autogroup
-r-------- 1 root root 0 Apr 26 08:49 auxv
-r--r--r-- 1 root root 0 Apr 26 08:49 cgroup
--w------- 1 root root 0 Apr 26 08:49 clear_refs
-r--r--r-- 1 root root 0 Apr 26 08:49 cmdline
-rw-r--r-- 1 root root 0 Apr 26 08:49 coredump_filter
-r--r--r-- 1 root root 0 Apr 26 08:49 cpuset
lrwxrwxrwx 1 root root 0 Apr 26 08:49 cwd
-r-------- 1 root root 0 Apr 26 08:49 environ
lrwxrwxrwx 1 root root 0 Apr 26 08:49 exe
dr-x------ 2 root root 0 Apr 26 08:49 fd
dr-x------ 2 root root 0 Apr 26 08:49 fdinfo
-r-------- 1 root root 0 Apr 26 08:49 io
-rw------- 1 root root 0 Apr 26 08:49 limits
-rw-r--r-- 1 root root 0 Apr 26 08:49 loginuid
-r--r--r-- 1 root root 0 Apr 26 08:49 maps
-rw------- 1 root root 0 Apr 26 08:49 mem
-r--r--r-- 1 root root 0 Apr 26 08:49 mountinfo
-r--r--r-- 1 root root 0 Apr 26 08:49 mounts
-r-------- 1 root root 0 Apr 26 08:49 mountstats
dr-xr-xr-x 5 customuser root 0 Apr 26 08:49 net
-r--r--r-- 1 root root 0 Apr 26 08:49 numa_maps
-rw-r--r-- 1 root root 0 Apr 26 08:49 oom_adj
-r--r--r-- 1 root root 0 Apr 26 08:49 oom_score
-rw-r--r-- 1 root root 0 Apr 26 08:49 oom_score_adj
-r--r--r-- 1 root root 0 Apr 26 08:49 pagemap
-r--r--r-- 1 root root 0 Apr 26 08:49 personality
lrwxrwxrwx 1 root root 0 Apr 26 08:49 root
-rw-r--r-- 1 root root 0 Apr 26 08:49 sched
-r--r--r-- 1 root root 0 Apr 26 08:49 schedstat
-r--r--r-- 1 root root 0 Apr 26 08:49 sessionid
-r--r--r-- 1 root root 0 Apr 26 08:49 smaps
-r--r--r-- 1 root root 0 Apr 26 08:49 stack
-r--r--r-- 1 root root 0 Apr 26 08:49 stat
-r--r--r-- 1 root root 0 Apr 26 08:49 statm
-r--r--r-- 1 root root 0 Apr 26 08:49 status
-r--r--r-- 1 root root 0 Apr 26 08:49 syscall
dr-xr-xr-x 14 customuser root 0 Apr 26 08:49 task
-r--r--r-- 1 root root 0 Apr 26 08:49 wchan

6

Sorry, I did mean pidof. A tool to query the PID of a process.

Alternatively ps aux filebeat | grep filebeat.

Having the pid we can read the proc directory of filebeat.

Is 13614 the pid of your filebeat instance? If so, how do you start filebeat?

7

@steffens

Yes, 13614 is the pid of my filebeat instance.

I start filebeat by using the initial script stored in /etc/init.d/filebeat .

PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

[ -f /etc/sysconfig/filebeat ] && . /etc/sysconfig/filebeat
pidfile=${PIDFILE-/var/run/filebeat.pid}
agent=${BEATS_AGENT-/usr/share/filebeat/bin/filebeat}
args="-c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat"
test_args="-e test config"
beat_user="${BEAT_USER:-customuser}"
wrapper="/usr/share/filebeat/bin/filebeat-god"
wrapperopts="-r / -n -p $pidfile"
user_wrapper="su"
user_wrapperopts="$beat_user -c"
RETVAL=0

# Source function library.
. /etc/rc.d/init.d/functions

# Determine if we can use the -p option to daemon, killproc, and status.
# RHEL < 5 can't.
if status | grep -q -- '-p' 2>/dev/null; then
    daemonopts="--pidfile $pidfile"
    pidopts="-p $pidfile"
fi

if command -v runuser >/dev/null 2>&1; then
    user_wrapper="runuser"
fi

[ "$beat_user" != "root" ] && wrapperopts="$wrapperopts -u $beat_user"

test() {
        $user_wrapper $user_wrapperopts "$agent $args $test_args"
}

start() {
    echo -n $"Starting filebeat: "
        test
        if [ $? -ne 0 ]; then
                echo
                exit 1
        fi
    daemon $daemonopts $wrapper $wrapperopts -- $agent $args
    RETVAL=$?
    echo
    return $RETVAL
}

stop() {
    echo -n $"Stopping filebeat: "
    killproc $pidopts $wrapper
    RETVAL=$?
    echo
    [ $RETVAL = 0 ] && rm -f ${pidfile}
}

restart() {
        test
        if [ $? -ne 0 ]; then
                return 1
        fi
    stop
    start
}

rh_status() {
    status $pidopts $wrapper
    RETVAL=$?
    return $RETVAL
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        start
    ;;
    stop)
        stop
    ;;
    restart)
        restart
    ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
    ;;
    status)
        rh_status
    ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart}"
        exit 1
esac

exit $RETVAL
8

I have the same issue. How can I fixed it?

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.