Elasticsearch performance issue

Elastic Stack Elasticsearch
1

Hello,

I had 2 node (version 2.x) test setup which was running properly. recently I have added two more nodes and changed the cluster name, due to cluster name change i have lost indexes and new indexes were created, finally the java API is taking 45 seconds to connect the cluster "Elasticsearch" and 40 sec to close the connection . The API simply collecting metric data of just 20 servers.
no logs were generated for slowlog index and slowlog search. How to sortout this delay delay?

Note:
nodes : es1,es2,es3 & es4 .
Nodes es1 & es2 are in public Ip range and es3 & es4 are in private Ip range.
Firewall is present between public Ip range and private ip range. so I have opened 9200,9300 for cluster communication through firewall.

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "Elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 43,
"active_shards" : 86,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

/etc/elasticsearch/elasticsearch.yml
cluster.name: Elasticsearch
script.inline: on
script.indexed: on
node.name: "localhost"
bootstrap.mlockall: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["es1:9300","es2:9300","es3:9300","es4:9300"]
indices.store.throttle.max_bytes_per_sec: 150mb
path.data: /elastic/data
path.repo: /elastic/backup
http.port: 9200
http.enabled: true
network.host: hostname
index.number_of_shards: 4
index.number_of_replicas: 1
node.master: true
node.data: true
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms
index.indexing.slowlog.level: info
index.indexing.slowlog.source: 1000

/etc/sysconfig/elasticsearch
ES_HEAP_SIZE=4g
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited

2

Don't set that. 2.X handles throttling automatically.[quote="rajkumar3v, post:1, topic:56974"]
changed the cluster name, due to cluster name change i have lost indexes and new indexes were created
[/quote]

The data will still be on disk.

What about the http API?

3

hello Mark Walkom,

  1. I have removed indices.store.throttle.max_bytes_per_sec: 150mb entry

  2. can i restore that indexes present is disk into new cluster?

  3. What about the http API? means, are you asking to test with Elastic head - Structured Query ?
    I have tested with elastic-head - Structured Query without any delay.

  4. I have corrected my post that java api is taking more time to create & close the connection with "Elasticsearch" cluster. not directly to index

4

Yes, take a look at Directory Layout | Elasticsearch Guide [2.3] | Elastic and then find your old directory. Shutdown the cluster and rename the directory to the new cluster name, then start ES. Of course if you do this you may lose the data that is under the new cluster, so take a backup.

Then it sounds like it's probably the code you are running

5

public static void main(String[] args) throws Exception {
TransportClient client=null;
try{

	    Settings settings = 
	      Settings.settingsBuilder().put("cluster.name", "Elasticsearch").put("client.transport.sniff", true).build();
	     client = new TransportClient.Builder().settings(settings).build();
	    client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("//localHost"), 9300));
		
		SearchResponse response = 
	        (SearchResponse)client.prepareSearch(new String[] { "IndexName" }).
			setTypes(new String[] { "typeName" }).
			setQuery(serverIDSearch).setFrom(0).setSize(1).
			addSort(timeMilliSort).execute().actionGet();
			
			
			 for (SearchHit hit : response.getHits())
	      {
	       
	      }
			    }catch(Exception e){
	    
			e.printStackTrace();
		}
	    finally {
	    	

	    	if(client!=null){
	    		
	    	

				System.out.println("startdatetime" + startdatetime);
	   	 client.close();
	 
	    	}
		}
}
	}
6

I have identified the issue,

nodes- es3 & es4 are behind the firewall so if i stop those cluster nodes, API is taking just 2 seconds to fetch the data from es1 & es2. then i started those 2 nodes and checked the time delay, it is 45 sec.
By default API query will send request to es3 or es4 node.

In firewall port 9200,9300 have opened for cluster communication. Is there any additional ports should be opened on Firewall? what would be the reason for this time delay issue?