617726909
(617726909)
February 20, 2017, 9:59am
1
Recently tested single and cluster insert performance is almost the same, I don't really know why? Use java API TransportClient Bulk Insert.
version:
ElasticSearch 5.1.1
One Index:
"settings" : {
"index" : {
"number_of_shards" : 5,
"number_of_replicas" : 0,
"refresh_interval" : "30s"
}
}
threads: 16 Bulk Insert:1000 docs(logs)
per log 44 field , per log for 1kb
single :
OS: Centos 7 8 cpus memery: 32GB
JVM Size Indexing Rate (/s)
4GB 6000~8000
8GB 6000~8000
16GB 6000~8000
cluster:
OS One: Centos 7 8 cpus memery: 32GB
master: JVM 4GB
date:JVM 16GB
OS Two: Centos 6.5 8 cpus memery: 32GB
date:JVM 16GB
Indexing Rate (/s) 6000~8000
I don't really know why?Bulk Insert docs performance is almost the same?
warkolm
(Mark Walkom)
February 20, 2017, 9:44pm
2
What does your client code look like?
617726909
(617726909)
February 21, 2017, 5:02am
3
Java API client code:
1: set clusterName、ipAndPorts
2:invoke initTransportClient method init connect
3:invoke bulkInsertDocument method,and set (index,type,json String list)
public class ElasticSearchBuilder {
private TransportClient client;
private String clusterName;
private String[] ipAndPorts;
public void setClusterName(String clusterName) {
this.clusterName = clusterName;
}
public void setIpAndPorts(String[] ipAndPorts) {
this.ipAndPorts = ipAndPorts;
}
public void initTransportClient() throws Exception {
if (client != null) {
client.close();
}
Settings settings = Settings.builder()
.put("cluster.name", clusterName)
.put("client.transport.sniff", true)
.build();
client = new PreBuiltTransportClient(settings);
if (ipAndPorts == null || ipAndPorts.length < 1) {
throw new NullPointerException("IP List is NULL");
}
for (String ipAndPort : ipAndPorts) {
String[] ipAndPortString = ipAndPort.split(":");
client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(ipAndPortString[0]), Integer.parseInt(ipAndPortString[1])));
}
}
public boolean bulkInsertDocument(String index, String type, List<String> jsonList) throws Exception {
BulkRequestBuilder bulkRequest = client.prepareBulk();
if (jsonList != null && jsonList.size() > 0) {
for (String jsonData : jsonList) {
bulkRequest.add(client.prepareIndex(index, type).setSource(jsonData));
}
}
BulkResponse bulkResponse = bulkRequest.get();
return !bulkResponse.hasFailures();
}
public void closeTransportClient() {
client.close();
}
}
system
(system)
Closed
March 21, 2017, 5:03am
4
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.