---
### Author:  [@Cara\_B](https://discuss.elastic.co/u/Cara_B)
#### Post date: [November 29, 2016, 3:14pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/5 "2016-11-29T15:14:58Z")
Thanks Martijn. I've updated this so it's a configurable value in our solution. I'll try a few different keep alive intervals and let you know how it goes.
I've been watching the TCP connections on node-1 and recording them at intervals over the last few hours. The ESTABLISHED connections between our ES nodes seem to stay consistent (the established connections have the same ports throughout the intervals) so it looks like they are persisting.
What strikes me as a bit strange are the connections in a state of TIME\_WAIT. There are usually around 300. They all have a local address of itself (the node-1 machine) and an iterating port number. The foreign address is also always the node-1 machine but port 9200. Each time I've recorded these stats (even when less than 2 minutes apart) some of these connections have vanished (closed?) and new ones (still iterating through the port numbers) have appeared. It looks like something on the ES node is making requests to it's own ES instance. **The thing of note is that the count of additional connections in TIME\_WAIT state is consistent with the increase in the count of HTTP connections in the ES stats.** Could something (Marvel?) be creating HTTP connections to the ES instance on the same machine and then the HTTP connection not being closed for some reason?
The total\_opened http connection count has now reached 1.3 million on node-1.
---
### Author:  [@Martijn\_Laarman](https://discuss.elastic.co/u/Martijn_Laarman)
#### Post date: [November 29, 2016, 3:32pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/6 "2016-11-29T15:32:39Z")
Mind posting your relevant marvel (exporter) settings ? Retract any credentials/real urls of course 🙂
---
### Author:  [@Martijn\_Laarman](https://discuss.elastic.co/u/Martijn_Laarman)
#### Post date: [November 29, 2016, 3:35pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/7 "2016-11-29T15:35:29Z")
Also on windows `netstat -aonb` will tell you which process owns which socket maybe there is some more information in there
---
### Author:  [@Cara\_B](https://discuss.elastic.co/u/Cara_B)
#### Post date: [November 29, 2016, 3:58pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/8 "2016-11-29T15:58:07Z")
I've been using the netstat -aonb command to export tcp connections - all the TIME\_WAIT connections have a PID of 0 so is of no help.... I've stopped Kibana on one of the machines and the total\_connections stopped increasing almost immediately.
Kibana settings are very basic - everything is commented out except:
_server.port: 5601_
_server.host: "123.45.67.890"_
_elasticsearch.url: "[http://123.45.67.890:9200](http://123.45.67.890:9200)"_
We have no Marvel configs in our ES config file. Could this be the issue?
---
### Author:  [@Martijn\_Laarman](https://discuss.elastic.co/u/Martijn_Laarman)
#### Post date: [November 29, 2016, 4:00pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/9 "2016-11-29T16:00:41Z")
No the default for marvel is to do a `local` export which does not use HTTP at all so that rules out marvel.
`123.45.67.890` is `node-1` ? That might explain why `node-1` is becoming a hotspot connection wise
---
### Author:  [@Cara\_B](https://discuss.elastic.co/u/Cara_B)
#### Post date: [November 29, 2016, 4:01pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/10 "2016-11-29T16:01:12Z")
That's just a dummy value - it's not the real one
---
### Author:  [@Martijn\_Laarman](https://discuss.elastic.co/u/Martijn_Laarman)
#### Post date: [November 29, 2016, 4:02pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/11 "2016-11-29T16:02:13Z")
good! question still stands though 🙂 is the `elasticsearch.url` configured in kibana pointing to `node-1`?
---
### Author:  [@Cara\_B](https://discuss.elastic.co/u/Cara_B)
#### Post date: [November 29, 2016, 4:11pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/12 "2016-11-29T16:11:22Z")
I turned Kibana off on node-2 as it's the one I need to monitor the least at the moment. All the nodes (including node-2) had an increase of total\_opened that was far too high. Therefore the elasticsearch.url is set in the Kibana.yml to point to node-2 ES e.g.
elasticsearch.yml
[node.name](http://node.name): node-2
network.host: 172.12.31.128
kibana.yml
server.port: 5601
server.host: "172.12.31.128"
elasticsearch.url: "[http://172.12.31.128:9200](http://172.12.31.128:9200)"
Let me know if you need more settings.
---
### Author:  [@Cara\_B](https://discuss.elastic.co/u/Cara_B)
#### Post date: [November 29, 2016, 4:20pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/13 "2016-11-29T16:20:42Z")
Also, do you know if the total\_opened value should drop? It isn't rising (now that Kibana is off) but it isn't dropping either.
---
### Author:  [@Martijn\_Laarman](https://discuss.elastic.co/u/Martijn_Laarman)
#### Post date: [November 29, 2016, 4:23pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/14 "2016-11-29T16:23:17Z")
No the total opened will not drop its a simple counter.
Are you able to restart the cluster and turn of kibana on all nodes and run your .NET application and see it behaves as intended?
That way the `total_opened` will reset and we can either rule out or focus on Kibana.
---
### Author:  [@Cara\_B](https://discuss.elastic.co/u/Cara_B)
#### Post date: [November 29, 2016, 4:37pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/15 "2016-11-29T16:37:13Z")
I've just turned Kibana off on node-1 too. Same result - the total\_opened value isn't increasing anymore. The JVM Heap usage also dropped instantly to a satisfactory percentage (from 81% to 52%).
It's pretty clearly Kibana. There must be something configured incorrectly but I'm just not sure what. Or is this behaving as it should?
---
### Author:  [@Martijn\_Laarman](https://discuss.elastic.co/u/Martijn_Laarman)
#### Post date: [November 29, 2016, 6:44pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/16 "2016-11-29T18:44:32Z")
Hi Cara
I will ping one of my Kibana colleagues to see if something obvious pops up for them.
For posterity and future googlers I'd like to explain why NEST chose 10k as a default connection limit.
In .NET `WebRequests` have a ServicePoint property assigned to them and requests sharing the same host have the same servicepoint. Through `ServicePointManager` you can set the max connections per servicepoint (or hostname by proxy).
`HttpWebRequest` will first try and reuse a socket that is open before trying to allocate more:
[https://referencesource.microsoft.com/#System/net/System/Net/\_ConnectionGroup.cs,305](https://referencesource.microsoft.com/#System/net/System/Net/_ConnectionGroup.cs,305)
In our performance testing we found that it was very hard to get the max concurrent connections to exceed more then 70-100 from a single box. So we even played with manually assigning `ServicePoint`'s from our own pool of `ServicePoint`'s to webrequest so that you could have 2 or 3 different `ServicePoints` per host.
While this did pump up the maximum concurrent open persistent connections to a node it did nothing for overall read and write throughput from a single box.
The idea leaving it at 10k was that HttpClient already saturates to a sane _true_ maximum state for concurrently open connections which might be different per machine.
I will leave the commit that dialed this back to a smaller constant so that it behaves more deterministic out of the box.
---
### Author:  [@spalger](https://discuss.elastic.co/u/spalger)
#### Post date: [November 29, 2016, 7:00pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/17 "2016-11-29T19:00:11Z")
@Cara_B this sounds like a long-standing issue from the JavaScript client that Kibana uses. [https://github.com/elastic/elasticsearch-js/issues/196](https://github.com/elastic/elasticsearch-js/issues/196)
I'll try to verify the issue and get a fix out, but unfortunately it will take some time before it's available in Kibana (maybe we can get it into 5.1)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 27, 2016, 7:00pm UTC](https://discuss.elastic.co/t/reusing-existing-http-connections/67130/18 "2016-12-27T19:00:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.