# Lsof shows "(deleted)" files
**URL:** https://discuss.elastic.co/t/lsof-shows-deleted-files/3444
**Category:** Elasticsearch
**Created:** [October 15, 2010, 4:14pm UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444 "2010-10-15T16:14:26Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@tommay](https://discuss.elastic.co/u/tommay)
#### Post date: [October 15, 2010, 4:14pm UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/1 "2010-10-15T16:14:26Z")
When I run lsof on the ElasticSearch java process it often shows that
some of the open files have been deleted. Does ElasticSearch and/or
Lucene really need to keep these deleted files open so it can continue
to use them, or is this just an oversight/bug?
E.g.,
$ lsof -p 6192 | egrep deleted | head -n 3
java 6192 gist 915r REG 202,2 103830 21147619
/home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.tis
(deleted)
java 6192 gist 918r REG 202,2 47849 21147621
/home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.frq
(deleted)
java 6192 gist 919r REG 202,2 24076 21147622
/home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.prx
(deleted)
Tom
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [October 15, 2010, 4:28pm UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/2 "2010-10-15T16:28:54Z")
Yes, those deleted files are needed to be kept around if there are open
search operations on going against them. Once they are done, those files
will be actually deleted (by the OS). Its really a file handle leak if you
see an increase of it over time.
-shay.banon
On Fri, Oct 15, 2010 at 6:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> When I run lsof on the Elasticsearch java process it often shows that
> some of the open files have been deleted. Does Elasticsearch and/or
> Lucene really need to keep these deleted files open so it can continue
> to use them, or is this just an oversight/bug?
>
> E.g.,
> $ lsof -p 6192 | egrep deleted | head -n 3
> java 6192 gist 915r REG 202,2 103830 21147619
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.tis
> (deleted)
> java 6192 gist 918r REG 202,2 47849 21147621
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.frq
> (deleted)
> java 6192 gist 919r REG 202,2 24076 21147622
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.prx
> (deleted)
>
> Tom
---
### Author:  [@tommay](https://discuss.elastic.co/u/tommay)
#### Post date: [October 15, 2010, 4:54pm UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/3 "2010-10-15T16:54:06Z")
That makes sense, but there are no open search operations on that
index, or any of the indexes, at least not any searches that were
initiated from an api. That's from a fresh run where I stopped ES,
deleted /work, restarted ES, and used the http bulk index API to put
data into a couple hundred indexes. I haven't done any searches yet,
I'm just poking around to see what kind of ulimit I might need. The
fds have been showing up as (deleted) for over an hour, and I'm
wondering if the process is ever going to close them.
I re-indexed the data into the same indexes and the fds were closed, so either:
- The files were closed properly by the application at some point.
- There is a leak, and the files were closed when java gc'd them.
Tom
On Fri, Oct 15, 2010 at 9:28 AM, Shay Banon
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> Yes, those deleted files are needed to be kept around if there are open
> search operations on going against them. Once they are done, those files
> will be actually deleted (by the OS). Its really a file handle leak if you
> see an increase of it over time.
> -shay.banon
>
> On Fri, Oct 15, 2010 at 6:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
>
> > When I run lsof on the Elasticsearch java process it often shows that
> > some of the open files have been deleted. Does Elasticsearch and/or
> > Lucene really need to keep these deleted files open so it can continue
> > to use them, or is this just an oversight/bug?
> >
> > E.g.,
> > $ lsof -p 6192 | egrep deleted | head -n 3
> > java 6192 gist 915r REG 202,2 103830 21147619
> >
> > /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.tis
> > (deleted)
> > java 6192 gist 918r REG 202,2 47849 21147621
> >
> > /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.frq
> > (deleted)
> > java 6192 gist 919r REG 202,2 24076 21147622
> >
> > /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.prx
> > (deleted)
> >
> > Tom
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [October 15, 2010, 5:03pm UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/4 "2010-10-15T17:03:27Z")
Internally, there is a scheduled refresh of the indices that might keep
those around (deleted files), and I not sure about the operating system
keeping some of them around and when it actually decides to deletes them.
There have been some tests run that shows that under load (both indexing and
searching) the number of opened files are not increased over time, but kept
at a steady number (+/- a delta). Usually, I recommend setting the max open
files to 16k or even 32k, as open files are also sockets and other.
If you do find in your tests that they keep increasing over time without
stopping, then I can have a look at it and fix it if there is a leak.
-shay.banon
On Fri, Oct 15, 2010 at 6:54 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> That makes sense, but there are no open search operations on that
> index, or any of the indexes, at least not any searches that were
> initiated from an api. That's from a fresh run where I stopped ES,
> deleted /work, restarted ES, and used the http bulk index API to put
> data into a couple hundred indexes. I haven't done any searches yet,
> I'm just poking around to see what kind of ulimit I might need. The
> fds have been showing up as (deleted) for over an hour, and I'm
> wondering if the process is ever going to close them.
>
> I re-indexed the data into the same indexes and the fds were closed, so
> either:
>
> - The files were closed properly by the application at some point.
> - There is a leak, and the files were closed when java gc'd them.
>
> Tom
>
> On Fri, Oct 15, 2010 at 9:28 AM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > Yes, those deleted files are needed to be kept around if there are open
> > search operations on going against them. Once they are done, those files
> > will be actually deleted (by the OS). Its really a file handle leak if
> > you
> > see an increase of it over time.
> > -shay.banon
> >
> > On Fri, Oct 15, 2010 at 6:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> >
> > > When I run lsof on the Elasticsearch java process it often shows that
> > > some of the open files have been deleted. Does Elasticsearch and/or
> > > Lucene really need to keep these deleted files open so it can continue
> > > to use them, or is this just an oversight/bug?
> > >
> > > E.g.,
> > > $ lsof -p 6192 | egrep deleted | head -n 3
> > > java 6192 gist 915r REG 202,2 103830 21147619
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.tis
>
> > > (deleted)
> > > java 6192 gist 918r REG 202,2 47849 21147621
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.frq
>
> > > (deleted)
> > > java 6192 gist 919r REG 202,2 24076 21147622
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.prx
>
> > > (deleted)
> > >
> > > Tom
---
### Author:  [@tommay](https://discuss.elastic.co/u/tommay)
#### Post date: [October 15, 2010, 10:13pm UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/5 "2010-10-15T22:13:42Z")
I'm not so concerned about the OS freeing the space. For completeness
I'll just mention that happens when a) the file has been removed
(i.e., no more hard links), and b) no process has the file open.
My concern was/is that java is removing the file but unintentionally
leaving it open, until gc calls a finalizer that closes it. In which
case gc tends to keep the number of these fds from increasing
indefinitely. I'm not saying that's happening, I don't have any
direct evidence, just that I've seen this before and it caught my eye
and made me wonder.
Tom
On Fri, Oct 15, 2010 at 10:03 AM, Shay Banon
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> Internally, there is a scheduled refresh of the indices that might keep
> those around (deleted files), and I not sure about the operating system
> keeping some of them around and when it actually decides to deletes them.
> There have been some tests run that shows that under load (both indexing and
> searching) the number of opened files are not increased over time, but kept
> at a steady number (+/- a delta). Usually, I recommend setting the max open
> files to 16k or even 32k, as open files are also sockets and other.
> If you do find in your tests that they keep increasing over time without
> stopping, then I can have a look at it and fix it if there is a leak.
> -shay.banon
>
> On Fri, Oct 15, 2010 at 6:54 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
>
> > That makes sense, but there are no open search operations on that
> > index, or any of the indexes, at least not any searches that were
> > initiated from an api. That's from a fresh run where I stopped ES,
> > deleted /work, restarted ES, and used the http bulk index API to put
> > data into a couple hundred indexes. I haven't done any searches yet,
> > I'm just poking around to see what kind of ulimit I might need. The
> > fds have been showing up as (deleted) for over an hour, and I'm
> > wondering if the process is ever going to close them.
> >
> > I re-indexed the data into the same indexes and the fds were closed, so
> > either:
> >
> > - The files were closed properly by the application at some point.
> > - There is a leak, and the files were closed when java gc'd them.
> >
> > Tom
> >
> > On Fri, Oct 15, 2010 at 9:28 AM, Shay Banon
> > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> >
> > > Yes, those deleted files are needed to be kept around if there are open
> > > search operations on going against them. Once they are done, those files
> > > will be actually deleted (by the OS). Its really a file handle leak if
> > > you
> > > see an increase of it over time.
> > > -shay.banon
> > >
> > > On Fri, Oct 15, 2010 at 6:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> > >
> > > > When I run lsof on the Elasticsearch java process it often shows that
> > > > some of the open files have been deleted. Does Elasticsearch and/or
> > > > Lucene really need to keep these deleted files open so it can continue
> > > > to use them, or is this just an oversight/bug?
> > > >
> > > > E.g.,
> > > > $ lsof -p 6192 | egrep deleted | head -n 3
> > > > java 6192 gist 915r REG 202,2 103830 21147619
> > > >
> > > > /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.tis
> > > > (deleted)
> > > > java 6192 gist 918r REG 202,2 47849 21147621
> > > >
> > > > /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.frq
> > > > (deleted)
> > > > java 6192 gist 919r REG 202,2 24076 21147622
> > > >
> > > > /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.prx
> > > > (deleted)
> > > >
> > > > Tom
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [October 16, 2010, 9:26am UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/6 "2010-10-16T09:26:24Z")
Hi,
As far as I know there is nothing like that in Lucene and it cleans up
once things are properly closed.
-shay.banon
On Sat, Oct 16, 2010 at 12:13 AM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> I'm not so concerned about the OS freeing the space. For completeness
> I'll just mention that happens when a) the file has been removed
> (i.e., no more hard links), and b) no process has the file open.
>
> My concern was/is that java is removing the file but unintentionally
> leaving it open, until gc calls a finalizer that closes it. In which
> case gc tends to keep the number of these fds from increasing
> indefinitely. I'm not saying that's happening, I don't have any
> direct evidence, just that I've seen this before and it caught my eye
> and made me wonder.
>
> Tom
>
> On Fri, Oct 15, 2010 at 10:03 AM, Shay Banon
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
>
> > Internally, there is a scheduled refresh of the indices that might keep
> > those around (deleted files), and I not sure about the operating system
> > keeping some of them around and when it actually decides to deletes them.
> > There have been some tests run that shows that under load (both indexing
> > and
> > searching) the number of opened files are not increased over time, but
> > kept
> > at a steady number (+/- a delta). Usually, I recommend setting the max
> > open
> > files to 16k or even 32k, as open files are also sockets and other.
> > If you do find in your tests that they keep increasing over time without
> > stopping, then I can have a look at it and fix it if there is a leak.
> > -shay.banon
> >
> > On Fri, Oct 15, 2010 at 6:54 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> >
> > > That makes sense, but there are no open search operations on that
> > > index, or any of the indexes, at least not any searches that were
> > > initiated from an api. That's from a fresh run where I stopped ES,
> > > deleted /work, restarted ES, and used the http bulk index API to put
> > > data into a couple hundred indexes. I haven't done any searches yet,
> > > I'm just poking around to see what kind of ulimit I might need. The
> > > fds have been showing up as (deleted) for over an hour, and I'm
> > > wondering if the process is ever going to close them.
> > >
> > > I re-indexed the data into the same indexes and the fds were closed, so
> > > either:
> > >
> > > - The files were closed properly by the application at some point.
> > > - There is a leak, and the files were closed when java gc'd them.
> > >
> > > Tom
> > >
> > > On Fri, Oct 15, 2010 at 9:28 AM, Shay Banon
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote:
> > >
> > > > Yes, those deleted files are needed to be kept around if there are
> > > > open
> > > > search operations on going against them. Once they are done, those
> > > > files
> > > > will be actually deleted (by the OS). Its really a file handle leak if
> > > > you
> > > > see an increase of it over time.
> > > > -shay.banon
> > > >
> > > > On Fri, Oct 15, 2010 at 6:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote:
> > > >
> > > > > When I run lsof on the Elasticsearch java process it often shows that
> > > > > some of the open files have been deleted. Does Elasticsearch and/or
> > > > > Lucene really need to keep these deleted files open so it can
> > > > > continue
> > > > > to use them, or is this just an oversight/bug?
> > > > >
> > > > > E.g.,
> > > > > $ lsof -p 6192 | egrep deleted | head -n 3
> > > > > java 6192 gist 915r REG 202,2 103830 21147619
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.tis
>
> > > > > (deleted)
> > > > > java 6192 gist 918r REG 202,2 47849 21147621
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.frq
>
> > > > > (deleted)
> > > > > java 6192 gist 919r REG 202,2 24076 21147622
>
> /home/tom/deploy/contacts/shared/work/contacts/nodes/0/indices/u30314/0/index/\_7.prx
>
> > > > > (deleted)
> > > > >
> > > > > Tom
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:17am UTC](https://discuss.elastic.co/t/lsof-shows-deleted-files/3444/7 "2017-07-06T04:17:57Z")