# Elasticsearch performance issue

**URL:** <https://discuss.elastic.co/t/elasticsearch-performance-issue/56974>\
**Category:** Elasticsearch\
**Created:** [August 2, 2016, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974 "2016-08-02T08:25:59Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rajkumar3v](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@rajkumar3v](https://discuss.elastic.co/u/rajkumar3v)\
**Post date:** [August 2, 2016, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/1 "2016-08-02T08:25:59Z")

</div>

Hello,

I had 2 node (version 2.x) test setup which was running properly. recently I have added two more nodes and changed the cluster name, due to cluster name change i have lost indexes and new indexes were created, finally the java API is taking 45 seconds to connect the cluster "Elasticsearch" and 40 sec to close the connection . The API simply collecting metric data of just 20 servers.  
no logs were generated for slowlog index and slowlog search. How to sortout this delay delay?

**Note:**  
nodes : es1,es2,es3 & es4 .  
Nodes es1 & es2 are in public Ip range and es3 & es4 are in private Ip range.  
Firewall is present between public Ip range and private ip range. so I have opened 9200,9300 for cluster communication through firewall.

curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)'  
{  
"cluster\_name" : "Elasticsearch",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 4,  
"number\_of\_data\_nodes" : 4,  
"active\_primary\_shards" : 43,  
"active\_shards" : 86,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 100.0  
}

**/etc/elasticsearch/elasticsearch.yml**  
[cluster.name](http://cluster.name): Elasticsearch  
script.inline: on  
script.indexed: on  
[node.name](http://node.name): "localhost"  
bootstrap.mlockall: true  
discovery.zen.ping.multicast.enabled: false  
discovery.zen.ping.unicast.hosts: ["es1:9300","es2:9300","es3:9300","es4:9300"]  
indices.store.throttle.max\_bytes\_per\_sec: 150mb  
path.data: /elastic/data  
path.repo: /elastic/backup  
http.port: 9200  
http.enabled: true  
network.host: hostname  
index.number\_of\_shards: 4  
index.number\_of\_replicas: 1  
node.master: true  
node.data: true  
index.search.slowlog.threshold.query.warn: 10s  
[index.search.slowlog.threshold.query.info](http://index.search.slowlog.threshold.query.info): 5s  
index.search.slowlog.threshold.query.debug: 2s  
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s  
[index.search.slowlog.threshold.fetch.info](http://index.search.slowlog.threshold.fetch.info): 800ms  
index.search.slowlog.threshold.fetch.debug: 500ms  
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.threshold.index.warn: 10s  
[index.indexing.slowlog.threshold.index.info](http://index.indexing.slowlog.threshold.index.info): 5s  
index.indexing.slowlog.threshold.index.debug: 2s  
index.indexing.slowlog.threshold.index.trace: 500ms  
index.indexing.slowlog.level: info  
index.indexing.slowlog.source: 1000

**/etc/sysconfig/elasticsearch**  
ES\_HEAP\_SIZE=4g  
ES\_STARTUP\_SLEEP\_TIME=5  
MAX\_OPEN\_FILES=65535  
MAX\_LOCKED\_MEMORY=unlimited

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 2, 2016, 10:00am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/2 "2016-08-02T10:00:32Z")

</div>

> [@rajkumar3v](#):
>
> indices.store.throttle.max\_bytes\_per\_sec: 150mb

Don't set that. 2.X handles throttling automatically.[quote="rajkumar3v, post:1, topic:56974"]  
changed the cluster name, due to cluster name change i have lost indexes and new indexes were created  
[/quote]

The data will still be on disk.

> [@rajkumar3v](#):
>
> finally the java API is taking 45 seconds to open the index and 40 seconds to close the indexes.

What about the http API?

---

<div class="post-metadata">

**Author:** ![rajkumar3v](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@rajkumar3v](https://discuss.elastic.co/u/rajkumar3v)\
**Post date:** [August 2, 2016, 10:43am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/3 "2016-08-02T10:43:49Z")

</div>

hello Mark Walkom,

1. I have removed indices.store.throttle.max\_bytes\_per\_sec: 150mb entry

2. can i restore that indexes present is disk into new cluster?

3. What about the http API? means, are you asking to test with Elastic head - Structured Query ?  
I have tested with elastic-head - Structured Query without any delay.

4. I have corrected my post that java api is taking more time to create & close the connection with "Elasticsearch" cluster. not directly to index

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 2, 2016, 10:45am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/4 "2016-08-02T10:45:55Z")

</div>

> [@rajkumar3v](#):
>
> can i restore that indexes present is disk into new cluster?

Yes, take a look at [Directory Layout | Elasticsearch Guide [2.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/setup-dir-layout.html) and then find your old directory. Shutdown the cluster and rename the directory to the new cluster name, then start ES. Of course if you do this you may lose the data that is under the new cluster, so take a backup.

> [@rajkumar3v](#):
>
> What about the http API? means, are you asking to test with Elastic head - Structured Query ? I have tested with elastic-head - Structured Query without any delay.

Then it sounds like it's probably the code you are running

---

<div class="post-metadata">

**Author:** ![rajkumar3v](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@rajkumar3v](https://discuss.elastic.co/u/rajkumar3v)\
**Post date:** [August 2, 2016, 11:16am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/5 "2016-08-02T11:16:28Z")

</div>

public static void main(String[] args) throws Exception {  
TransportClient client=null;  
try{

```
	    Settings settings = 
	      Settings.settingsBuilder().put("cluster.name", "Elasticsearch").put("client.transport.sniff", true).build();
	     client = new TransportClient.Builder().settings(settings).build();
	    client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("//localHost"), 9300));
		
		SearchResponse response = 
	        (SearchResponse)client.prepareSearch(new String[] { "IndexName" }).
			setTypes(new String[] { "typeName" }).
			setQuery(serverIDSearch).setFrom(0).setSize(1).
			addSort(timeMilliSort).execute().actionGet();
			
			
			 for (SearchHit hit : response.getHits())
	      {
	       
	      }
			    }catch(Exception e){
	    
			e.printStackTrace();
		}
	    finally {
	    	

	    	if(client!=null){
	    		
	    	

				System.out.println("startdatetime" + startdatetime);
	   	 client.close();
	 
	    	}
		}
}
	}
```

---

<div class="post-metadata">

**Author:** ![rajkumar3v](https://avatars.discourse-cdn.com/v4/letter/r/73ab20/32.png) [@rajkumar3v](https://discuss.elastic.co/u/rajkumar3v)\
**Post date:** [August 3, 2016, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/6 "2016-08-03T08:15:17Z")

</div>

I have identified the issue,

nodes- es3 & es4 are behind the firewall so if i stop those cluster nodes, API is taking just 2 seconds to fetch the data from es1 & es2. then i started those 2 nodes and checked the time delay, it is 45 sec.  
By default API query will send request to es3 or es4 node.

In firewall port 9200,9300 have opened for cluster communication. Is there any additional ports should be opened on Firewall? what would be the reason for this time delay issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:30pm UTC](https://discuss.elastic.co/t/elasticsearch-performance-issue/56974/7 "2017-07-05T22:30:29Z")

</div>


