How to maintain ES connection in web application

Elastic Stack Elasticsearch
1

Hey guys,

I am using ES as the back end of my search application. When my web application starts, I start a new transport client and keep it alive. It works well in the first few hours/days. But after a while, it would crash and the web log give me a leak of memory error.

I think this is a common task. I was wondering if there is already a solution to this.

Thanks,
Cody

2

Hi Cody,

I am not sure about the programming language about your back end, we use java and HTTP connection pool. We have been using it for more than 2 years and haven't seen any issues. Connection pool also allows us to shield Elastic from surge in traffic as well, without connection pool we were reaching the limit of open network connections.

Cheers,
Jinal

3

I am using Java too. What is the HTTP connection pool you are talking about, is it some third-party library?

What I did is create a transport client in servlet web listener, and all of the users share the same transport client. I know this is not a good way, but we only have less than 5 users in the testbed.

Can you talk a bit more about how you handle this? I know some people user Hibernate to manage Database connection, but I am not sure about the ES case. Thanks a lot!

4

We use Apache HTTP library. Implementation was using PoolingClientConnectionManager which is now deprecated so I would suggest using PoolingHttpClientConnectionManager.

5

Thanks. One more question, how about ES connection? Did you just create one client and leave it open all the time?

6

We don't use ES Java SDK. We manage creating json document and queries and use HTTP to talk to ES.

7

One of our recent project is using Jest to connect to ES, underneath Jest uses the HTTP connection pooling.

Reference: https://github.com/searchbox-io/Jest/blob/master/jest/src/main/java/io/searchbox/client/JestClientFactory.java

8

Yeah, thanks, I just saw JEST on Github. Using HTTP client sounds like a good solution, but it also requires to rewrite everything about ES operation in the code, which is a lot of work. Wonder if there is any workaround using transport client.

9

You can also look at the official http client which is coming in 5.0. I tested it against 1.x, 2.x and 5.x.
Note that it's a low level one.