# CLOSE\_WAIT Sockets **URL:** \ **Category:** Elasticsearch\ **Created:** [November 11, 2010, 6:02pm UTC](https://discuss.elastic.co/t/close-wait-sockets/3558 "2010-11-11T18:02:59Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![elasticsearcher](https://avatars.discourse-cdn.com/v4/letter/e/7ba0ec/32.png) [@elasticsearcher](https://discuss.elastic.co/u/elasticsearcher)\ **Post date:** [November 11, 2010, 6:02pm UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/1 "2010-11-11T18:02:59Z")
Hi all, I'm running ElasticSearch backing up to HDFS with the following elasticsearch.yml: index: store: fs: memory: enabled: true gateway: type: hdfs hdfs: uri: hdfs://blah:54310 path: elasticsearch/gateway I've noticed that over the last week the number of sockets in the CLOSE\_WAIT status has grown steadily. $ netstat -aonp | grep CLOSE\_WAIT | wc -l 6484 When I look at the sockets, I can see the process id is elasticsearch and the destination port is 50010, which I looked up and is a hadoop datanode port. Has anyone seen this before/have any ideas about how to fix this? Thanks! ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [November 12, 2010, 1:07am UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/2 "2010-11-12T01:07:52Z")
There are settings to reduce the amount of time that a socket stays in this state after closing. I know on Windows this is 2 minutes and is reg configurable. Not sure about other OSes. However, sockets should get reused. What client are you using and if it is rest/HTTP based, is it using keep-alive? On Nov 11, 11:02 am, elasticsearcher [elasticsearc...@gmail.com](mailto:elasticsearc...@gmail.com) wrote: > Hi all, > > I'm running Elasticsearch backing up to HDFS with the following > elasticsearch.yml: > > index: > store: > fs: > memory: > enabled: true > gateway: > type: hdfs > hdfs: > uri: hdfs://blah:54310 > path: elasticsearch/gateway > > I've noticed that over the last week the number of sockets in the CLOSE\_WAIT > status has grown steadily. > > $ netstat -aonp | grep CLOSE\_WAIT | wc -l > 6484 > > When I look at the sockets, I can see the process id is elasticsearch and > the destination port is 50010, which I looked up and is a hadoop datanode > port. > > Has anyone seen this before/have any ideas about how to fix this? > > ## Thanks! > > View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/CLOSE-WAIT-Sockets-tp](http://elasticsearch-users.115913.n3.nabble.com/CLOSE-WAIT-Sockets-tp)... > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 12, 2010, 5:54am UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/3 "2010-11-12T05:54:27Z")
Hi, I am not doing anything specific on hadoop except for using the formal API. I would think it would reuse the same socket when talking to the hadoop cluster... . Maybe you could check a bit with hadoop and see why it might happen? -shay.banon On Fri, Nov 12, 2010 at 3:07 AM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > There are settings to reduce the amount of time that a socket stays in > this state after closing. I know on Windows this is 2 minutes and is > reg configurable. Not sure about other OSes. > > However, sockets should get reused. What client are you using and if > it is rest/HTTP based, is it using keep-alive? > > On Nov 11, 11:02 am, elasticsearcher [elasticsearc...@gmail.com](mailto:elasticsearc...@gmail.com) > wrote: > > > Hi all, > > > > I'm running Elasticsearch backing up to HDFS with the following > > elasticsearch.yml: > > > > index: > > store: > > fs: > > memory: > > enabled: true > > gateway: > > type: hdfs > > hdfs: > > uri: hdfs://blah:54310 > > path: elasticsearch/gateway > > > > I've noticed that over the last week the number of sockets in the > > CLOSE\_WAIT > > status has grown steadily. > > > > $ netstat -aonp | grep CLOSE\_WAIT | wc -l > > 6484 > > > > When I look at the sockets, I can see the process id is elasticsearch and > > the destination port is 50010, which I looked up and is a hadoop datanode > > port. > > > > Has anyone seen this before/have any ideas about how to fix this? > > > > ## Thanks! > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/CLOSE-WAIT-Sockets-tp](http://elasticsearch-users.115913.n3.nabble.com/CLOSE-WAIT-Sockets-tp)... > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![elasticsearcher](https://avatars.discourse-cdn.com/v4/letter/e/7ba0ec/32.png) [@elasticsearcher](https://discuss.elastic.co/u/elasticsearcher)\ **Post date:** [November 12, 2010, 10:56pm UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/4 "2010-11-12T22:56:11Z")
Our base client is based on the pyelasticsearch class found on the elasticsearch website. I've looked into this and python uses keep-alive by default in the client as far as I can tell. I'm pretty sure that it isn't the client that is the problem since the port numbers indicate that it is a problem between hadoop and elasticsearch. I'll take a look at the interface between the two and see if I can find anything, if not I'll try to find another way around it. Thanks. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 13, 2010, 12:30am UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/5 "2010-11-13T00:30:24Z")
Yea, it looks like a CLOSED\_WAIT communicating with hadoop. In elasticsearch, a single FileSystem instance is used for the node, not sure how hadoop internals work to tell how it manages sockets. Might be the expected behavior? On Sat, Nov 13, 2010 at 12:56 AM, elasticsearcher \<[elasticsearcher@gmail.com](mailto:elasticsearcher@gmail.com) > wrote: > Our base client is based on the pyelasticsearch class found on the > elasticsearch website. I've looked into this and python uses keep-alive by > default in the client as far as I can tell. I'm pretty sure that it isn't > the client that is the problem since the port numbers indicate that it is a > problem between hadoop and elasticsearch. I'll take a look at the interface > between the two and see if I can find anything, if not I'll try to find > another way around it. > > ## Thanks. > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/CLOSE-WAIT-Sockets-tp1884117p1892000.html](http://elasticsearch-users.115913.n3.nabble.com/CLOSE-WAIT-Sockets-tp1884117p1892000.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![elasticsearcher](https://avatars.discourse-cdn.com/v4/letter/e/7ba0ec/32.png) [@elasticsearcher](https://discuss.elastic.co/u/elasticsearcher)\ **Post date:** [November 15, 2010, 6:53pm UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/6 "2010-11-15T18:53:23Z")
I've found something interesting: $ ps aux | grep elasticsearch \<10600\> $ /usr/sbin/lsof -p 10600 | grep CLOSE\_WAIT \ $ netstat -lp | grep 50010 \<27091\> $ /usr/sbin/lsof -p 27091 | grep CLOSE\_WAIT Confirming: $ /usr/sbin/lsof | grep CLOSE\_WAIT | grep -v 10600 It appears that something is wrong with the way elasticsearch is using the hadoop api or something is wrong with the hadoop api, since I wouldn't think having thousands of CLOSE\_WAIT sockets would be expected behavior. I'll keep looking and maybe post on the hadoop forums. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:16am UTC](https://discuss.elastic.co/t/close-wait-sockets/3558/7 "2017-07-06T04:16:29Z")