# Filebeat: Error while retrieving FD information: error getting number of open FD: key not found

**URL:** <https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 25, 2019, 6:56am UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378 "2019-04-25T06:56:00Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kutomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kutomi/32/44984_2.png) [@kutomi](https://discuss.elastic.co/u/kutomi)\
**Post date:** [April 25, 2019, 6:56am UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/1 "2019-04-25T06:56:01Z")

</div>

> [@Filebeat: Error on FD](https://discuss.elastic.co/t/filebeat-error-on-fd/173616):
>
> We recently started using filebeat and its on latest version 6.6 After initiating filebeat daemon process, we are able to push messages to kafka topics Also we started seeing some errors around the metric data in the logs ERROR instance/metrics\_file\_descriptors.go:39 Error while retrieving FD information: error getting number of open FD: key not found Any idea what all things I should be considered to debug further? How to ensure such errors doesnt occur again in future?

I also got the same error as this post.  
`ERROR instance/metrics_file_descriptors.go:39 Error while retrieving FD information: error getting number of open FD: key not found`

I wonder what is the cause of this error?  
And, I also want to know how could I solve it?

Filebeat Version: filebeat version 6.5.4 (amd64), libbeat 6.5.4  
OS Version: CentOS release 6.3 (Final)

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 25, 2019, 10:54am UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/2 "2019-04-25T10:54:16Z")

</div>

On linux filebeat tries to get the number of file descriptors from the `/proc/<pid>/fd` file. The error indicates that file descriptor usage is not available in or the file can not be read by the process.

---

<div class="post-metadata">

**Author:** ![kutomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kutomi/32/44984_2.png) [@kutomi](https://discuss.elastic.co/u/kutomi)\
**Post date:** [April 25, 2019, 11:57pm UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/3 "2019-04-25T23:57:26Z")

</div>

@steffens

I have confirmed the permission `/proc/<pid>/fd` is as below.  
`dr-x------ 2 root root 0 Apr 26 08:49 fd`

I am executing Filebeat by using custom user rather than root. I don't think the process has permission to read the file.

In order to solve it, what should I do?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 26, 2019, 3:21pm UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/4 "2019-04-26T15:21:38Z")

</div>

can you run:

```auto
$ psof filebeat

and 

$ ls -la /proc/$(psof filebeat)/ 

```

I'd assume filebeat should be able to read it's own state. Actually it would be better if filebeat did read `/proc/self`.

---

<div class="post-metadata">

**Author:** ![kutomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kutomi/32/44984_2.png) [@kutomi](https://discuss.elastic.co/u/kutomi)\
**Post date:** [May 6, 2019, 11:57pm UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/5 "2019-05-06T23:57:47Z")

</div>

@steffens

Sorry for the late response.  
I cannot find the command of `psof` in my server nor Google.  
Do you mean `lsof` instead?

An error is occurred when I executed `lsof`

> lsof: WARNING: can't stat() devtmpfs file system /var/unbound/dev/log  
> Output information may be incomplete.  
> lsof: WARNING: can't stat() devtmpfs file system /var/unbound/dev/random  
> Output information may be incomplete.  
> lsof: status error on filebeat: No such file or directory

For the result of `ls -la /proc/<pid>` is as below

> ls: cannot read symbolic link /proc/13614/cwd: Permission denied  
> ls: cannot read symbolic link /proc/13614/root: Permission denied  
> ls: cannot read symbolic link /proc/13614/exe: Permission denied  
> total 0  
> dr-xr-xr-x 7 customuser root 0 Apr 26 08:49 .  
> dr-xr-xr-x 173 root root 0 Dec 7 2017 ..  
> dr-xr-xr-x 2 customuser root 0 Apr 26 08:49 attr  
> -rw-r--r-- 1 root root 0 Apr 26 08:49 autogroup  
> -r-------- 1 root root 0 Apr 26 08:49 auxv  
> -r--r--r-- 1 root root 0 Apr 26 08:49 cgroup  
> --w------- 1 root root 0 Apr 26 08:49 clear\_refs  
> -r--r--r-- 1 root root 0 Apr 26 08:49 cmdline  
> -rw-r--r-- 1 root root 0 Apr 26 08:49 coredump\_filter  
> -r--r--r-- 1 root root 0 Apr 26 08:49 cpuset  
> lrwxrwxrwx 1 root root 0 Apr 26 08:49 cwd  
> -r-------- 1 root root 0 Apr 26 08:49 environ  
> lrwxrwxrwx 1 root root 0 Apr 26 08:49 exe  
> dr-x------ 2 root root 0 Apr 26 08:49 fd  
> dr-x------ 2 root root 0 Apr 26 08:49 fdinfo  
> -r-------- 1 root root 0 Apr 26 08:49 io  
> -rw------- 1 root root 0 Apr 26 08:49 limits  
> -rw-r--r-- 1 root root 0 Apr 26 08:49 loginuid  
> -r--r--r-- 1 root root 0 Apr 26 08:49 maps  
> -rw------- 1 root root 0 Apr 26 08:49 mem  
> -r--r--r-- 1 root root 0 Apr 26 08:49 mountinfo  
> -r--r--r-- 1 root root 0 Apr 26 08:49 mounts  
> -r-------- 1 root root 0 Apr 26 08:49 mountstats  
> dr-xr-xr-x 5 customuser root 0 Apr 26 08:49 net  
> -r--r--r-- 1 root root 0 Apr 26 08:49 numa\_maps  
> -rw-r--r-- 1 root root 0 Apr 26 08:49 oom\_adj  
> -r--r--r-- 1 root root 0 Apr 26 08:49 oom\_score  
> -rw-r--r-- 1 root root 0 Apr 26 08:49 oom\_score\_adj  
> -r--r--r-- 1 root root 0 Apr 26 08:49 pagemap  
> -r--r--r-- 1 root root 0 Apr 26 08:49 personality  
> lrwxrwxrwx 1 root root 0 Apr 26 08:49 root  
> -rw-r--r-- 1 root root 0 Apr 26 08:49 sched  
> -r--r--r-- 1 root root 0 Apr 26 08:49 schedstat  
> -r--r--r-- 1 root root 0 Apr 26 08:49 sessionid  
> -r--r--r-- 1 root root 0 Apr 26 08:49 smaps  
> -r--r--r-- 1 root root 0 Apr 26 08:49 stack  
> -r--r--r-- 1 root root 0 Apr 26 08:49 stat  
> -r--r--r-- 1 root root 0 Apr 26 08:49 statm  
> -r--r--r-- 1 root root 0 Apr 26 08:49 status  
> -r--r--r-- 1 root root 0 Apr 26 08:49 syscall  
> dr-xr-xr-x 14 customuser root 0 Apr 26 08:49 task  
> -r--r--r-- 1 root root 0 Apr 26 08:49 wchan

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 8, 2019, 11:49am UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/6 "2019-05-08T11:49:02Z")

</div>

Sorry, I did mean `pidof`. A tool to query the PID of a process.

Alternatively `ps aux filebeat | grep filebeat`.

Having the pid we can read the proc directory of filebeat.

Is `13614` the pid of your filebeat instance? If so, how do you start filebeat?

---

<div class="post-metadata">

**Author:** ![kutomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kutomi/32/44984_2.png) [@kutomi](https://discuss.elastic.co/u/kutomi)\
**Post date:** [May 9, 2019, 12:16am UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/7 "2019-05-09T00:16:15Z")

</div>

@steffens

Yes, `13614` is the pid of my filebeat instance.

I start filebeat by using the initial script stored in `/etc/init.d/filebeat` .

```
PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

[-f /etc/sysconfig/filebeat] && . /etc/sysconfig/filebeat
pidfile=${PIDFILE-/var/run/filebeat.pid}
agent=${BEATS_AGENT-/usr/share/filebeat/bin/filebeat}
args="-c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat"
test_args="-e test config"
beat_user="${BEAT_USER:-customuser}"
wrapper="/usr/share/filebeat/bin/filebeat-god"
wrapperopts="-r / -n -p $pidfile"
user_wrapper="su"
user_wrapperopts="$beat_user -c"
RETVAL=0

# Source function library.
. /etc/rc.d/init.d/functions

# Determine if we can use the -p option to daemon, killproc, and status.
# RHEL < 5 can't.
if status | grep -q -- '-p' 2>/dev/null; then
    daemonopts="--pidfile $pidfile"
    pidopts="-p $pidfile"
fi

if command -v runuser >/dev/null 2>&1; then
    user_wrapper="runuser"
fi

["$beat_user" != "root"] && wrapperopts="$wrapperopts -u $beat_user"

test() {
        $user_wrapper $user_wrapperopts "$agent $args $test_args"
}

start() {
    echo -n $"Starting filebeat: "
        test
        if [$? -ne 0]; then
                echo
                exit 1
        fi
    daemon $daemonopts $wrapper $wrapperopts -- $agent $args
    RETVAL=$?
    echo
    return $RETVAL
}

stop() {
    echo -n $"Stopping filebeat: "
    killproc $pidopts $wrapper
    RETVAL=$?
    echo
    [$RETVAL = 0] && rm -f ${pidfile}
}

restart() {
        test
        if [$? -ne 0]; then
                return 1
        fi
    stop
    start
}

rh_status() {
    status $pidopts $wrapper
    RETVAL=$?
    return $RETVAL
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        start
    ;;
    stop)
        stop
    ;;
    restart)
        restart
    ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
    ;;
    status)
        rh_status
    ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart}"
        exit 1
esac

exit $RETVAL
```

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [May 14, 2019, 12:42pm UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/8 "2019-05-14T12:42:23Z")

</div>

I have the same issue. How can I fixed it?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2019, 12:42pm UTC](https://discuss.elastic.co/t/filebeat-error-while-retrieving-fd-information-error-getting-number-of-open-fd-key-not-found/178378/9 "2019-06-11T12:42:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.