# Connection pooling in Python

**URL:** https://discuss.elastic.co/t/connection-pooling-in-python/189725
**Category:** Elasticsearch
**Created:** [July 10, 2019, 10:46am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725 "2019-07-10T10:46:56Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![ssharma7884](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssharma7884/32/45343_2.png) [@ssharma7884](https://discuss.elastic.co/u/ssharma7884)
#### Post date: [July 10, 2019, 10:46am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/1 "2019-07-10T10:46:56Z")

</div>

Hi,  
I am using python to connect to elastic search DB cluster.  
No. of nodes = 3, each node has a user id and password

I am trying to implement connection pooling, referred the documentation for python api at:  
[https://elasticsearch-py.readthedocs.io/en/master/connection.html](https://elasticsearch-py.readthedocs.io/en/master/connection.html)  
...under the " Connection Pool" section...but could not get it to work.

Can someone please point me to a working example to implement connection pooling using python

Thanks & Regards,  
Sachin

---

<div class="post-metadata">

### Author: ![ssharma7884](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssharma7884/32/45343_2.png) [@ssharma7884](https://discuss.elastic.co/u/ssharma7884)
#### Post date: [July 18, 2019, 8:46am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/2 "2019-07-18T08:46:06Z")

</div>

I got this to work...sample code below....

from elasticsearch import Transport

try:  
qry = {"query": {"bool": {"must": [{"match": {"extension": "css"}}, {"match": {"machine.os": "ios"}}]}}}

```
transport = Transport([{'host': '<your_host1>'}, {'host': '<your_host2>'}], http_auth=('<your_user>', '<user_password>'))
print('no of connections in connection pool before adding connection = ' + str(len(transport.connection_pool.connections)))
#you may increase the range value from 2 to required no. as per your requirement
for cntr in range(2):
	transport.add_connection({'host': '<your_host1>'})
	transport.add_connection({'host': '<your_host2>'})
print('no of connections in connection pool after adding connection = ' + str(len(transport.connection_pool.connections)))
cnn1 = transport.get_connection()
cnn2 = transport.get_connection()
cnn3 = transport.get_connection()
cnn4 = transport.get_connection()
cnn5 = transport.get_connection()
cnn6 = transport.get_connection()
cnn7 = transport.get_connection()
cnn8 = transport.get_connection()
cnn9 = transport.get_connection()
cnn10 = transport.get_connection()

result = transport.perform_request(method='GET', url='/<index_name_to_query>/_search', body=qry)
for doc in result['hits']['hits']:
	print('got data...')

```

except Exception as e:  
print('exception...')  
print(str(e))

---

<div class="post-metadata">

### Author: ![bry-c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bry-c/32/49123_2.png) [@bry-c](https://discuss.elastic.co/u/bry-c)
#### Post date: [July 19, 2019, 4:28am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/3 "2019-07-19T04:28:47Z")

</div>

Hi @ssharma7884,

You do not need to call directly the Transport class.

You can just initialize the Elaticsearch class and you can access all functions.

Check the link to see examples:  
[https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch](https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch)

---

<div class="post-metadata">

### Author: ![ssharma7884](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssharma7884/32/45343_2.png) [@ssharma7884](https://discuss.elastic.co/u/ssharma7884)
#### Post date: [July 19, 2019, 6:50am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/4 "2019-07-19T06:50:50Z")

</div>

Hi bry-c,  
Thanks for the link.  
I was initially using the Elasticsearch class as below:

from elasticsearch import Elasticsearch  
....  
elastic\_conn = Elasticsearch(['\<my\_host\>'], http\_auth=('\<my\_user\>', '\<my\_user\_pwd\>'))  
qry = {"query": {"bool": {"must": [{"match": {"extension": "css"}}, {"match": {"machine.os": "ios"}}]}}}  
res = elastic\_conn.search(index="\<my\_index\_name\>", body=qry)  
for doc in res['hits']['hits']:  
print('got data...')

But, I need to implement connection pooling at server start up & use a connection from this pool (by using --\> transport.get\_connection()) for all future queries. I could not find a link to implement connection pooling using the above, hence did connection pooling using the "Transport" class.

How can I implement connection pooling (at server start up) as per your link?

Thanks & Regards,  
Sachin

---

<div class="post-metadata">

### Author: ![bry-c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bry-c/32/49123_2.png) [@bry-c](https://discuss.elastic.co/u/bry-c)
#### Post date: [July 19, 2019, 8:37am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/5 "2019-07-19T08:37:30Z")

</div>

Hi @ssharma7884,

If you use django you can create one module inside your project folder.

```
# es_conn.py
import elasticsearch
from django.conf import settings

es = elasticsearch.Elasticsearch(settings.ELASTIC_SERVER, **settings.ELASTIC_CONFIG)

```

The es instance will only initialize once on start up.  
Elasticsearch instance already create a pool of connection from the provided array of hosts.

Then you can use it like this in other module.

```
# other_module.py
from es_conn import es

es.exists(index='index_name', doc_type='doc', id='123')
```

---

<div class="post-metadata">

### Author: ![ssharma7884](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssharma7884/32/45343_2.png) [@ssharma7884](https://discuss.elastic.co/u/ssharma7884)
#### Post date: [July 21, 2019, 8:31am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/6 "2019-07-21T08:31:12Z")

</div>

bry-c,  
Thanks for your comments.  
I am using flask & apache2. My existing connection pooling implementation is similar to what you have mentioned.  
After your reply, I re-read the following links/sections which gave me the required details & clarifications I had missed out initially

[https://elasticsearch-py.readthedocs.io/en/master/#persistent-connections](https://elasticsearch-py.readthedocs.io/en/master/#persistent-connections)  
`elasticsearch-py` uses **persistent connections inside of individual connection pools (one per each configured or sniffed node)**

[https://elasticsearch-py.readthedocs.io/en/master/#thread-safety](https://elasticsearch-py.readthedocs.io/en/master/#thread-safety)  
**By default we allow `urllib3` to open up to 10 connections to each node, if your application calls for more parallelism, use the `maxsize` parameter to raise the limit:**

**#maxsize parameter for connection poolsize**  
es = Elasticsearch(["host1", "host2"], maxsize=25)

Query

1. Here, I am assuming that the parameter "maxsize" is for no. of persistent connections per confirgured node.
2. As per the thread-safety link above :  
If your application is long-running consider turning on [Sniffing](https://elasticsearch-py.readthedocs.io/en/master/#sniffing) to make sure the client is up to date on the cluster location.

My application is long running almost 24x7, so should I turn on sniffing mechanism?

Thanks & Regards,  
Sachin Vyas.

---

<div class="post-metadata">

### Author: ![ssharma7884](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssharma7884/32/45343_2.png) [@ssharma7884](https://discuss.elastic.co/u/ssharma7884)
#### Post date: [July 21, 2019, 8:46am UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/7 "2019-07-21T08:46:37Z")

</div>

I also came across the following 2 links, which provided further clarification:

[https://elasticsearch-py.readthedocs.io/en/master/connection.html#elasticsearch.Urllib3HttpConnection](https://elasticsearch-py.readthedocs.io/en/master/connection.html#elasticsearch.Urllib3HttpConnection)

Refer the "maxsize" parameter comments under the above link & also the link:  
[https://urllib3.readthedocs.io/en/1.4/pools.html#api](https://urllib3.readthedocs.io/en/1.4/pools.html#api)

Thanks & Regards,  
Sachin Vyas.

---

<div class="post-metadata">

### Author: ![bry-c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bry-c/32/49123_2.png) [@bry-c](https://discuss.elastic.co/u/bry-c)
#### Post date: [July 21, 2019, 12:45pm UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/8 "2019-07-21T12:45:45Z")

</div>

Hi @ssharma7884,

Yes it’s better to enable sniffing as mentioned in the documentation.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 18, 2019, 12:45pm UTC](https://discuss.elastic.co/t/connection-pooling-in-python/189725/9 "2019-08-18T12:45:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.