I want to improve the search speed

Current problem, the search speed is slow.
About the system under specification and set lists.

System Specifications
Number of nodes: 4
CPU model: Quad-Core AMD Opteron(tm) Processor 2347 HE (1900 MHz)
CPU total logical cores: 8
Memory : 8G

Elasticsearch Settings (Elasticsearch version is 0.20.6)
index.number_of_shards: 8
index.number_of_replicas: 1
node.master: true
node.data: true
bootstrap.mlockall: true
transport.tcp.compress: true
network.tcp.keep_alive: true
cluster.routing.allocation.node_initial_primaries_recoveries: 1
cluster.routing.allocation.node_concurrent_recoveries: 1
cluster.routing.allocation.cluster_concurrent_rebalance: 1
index.shard.recovery.concurrent_streams: 2
cluster.routing.allocation.disable_allocation: false
cluster.routing.allocation.disable_replica_allocation: false
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
index.cache.field.type: soft
index.term_index_interval: 64
index.refresh_interval: 30

Elasticsearch Start option
bin/elasticsearch -Xmx4G -Xms4G -Des.index.storage.type=memory

Documents Info
size: 49gb (98.2gb)
docs: 18214105 (18214590)

Search Condition

  1. Were routing to each user ID.
  2. Java Code
    FilterBuilder[] reporterFilters = null;
    FilterBuilder filters = null;

SearchRequestBuilder builder = client.prepareSearch(index);
builder.setFrom((Integer) params.get("searchFrom"));
builder.setSize((Integer) params.get("searchCnt"));
builder.setTypes(type);
builder.addSort((String) params.get("sortField"), (Boolean) params.get("ascending") ? SortOrder.ASC : SortOrder.DESC);

//Set date range filter
if (params.get("fromDate") != null) {
filters = FilterBuilders.rangeFilter("regDate").from(params.get("fromDate")).to(params.get("toDate"));
}

//Set category filter
if (params.get("categoryId") != null) {
filters = FilterBuilders.andFilter(FilterBuilders.termFilter("categoryId", params.get("categoryId")), filters);
}

//Set query filter
if (params.get("query") != null) {
builder.setQuery(QueryBuilders
.queryString((String) params.get("query"))
.field("title")
.field("summary")
.field("content")
.defaultOperator(QueryStringQueryBuilder.Operator.AND)
.analyzer("cjk")
.useDisMax(true)
);
}

builder.setFilter(filters);

SearchResponse response = builder
.execute()
.actionGet();

Connector Code

@Component
public class ElasticsearchConnector {
@Autowired
PropertiesUtil propertiesUtil;
public TransportClient client;

@PostConstruct
private void initElasticSearch() {
    Settings settings = ImmutableSettings.settingsBuilder()
            .put("cluster.name", propertiesUtil.elasticsearchClusterName)
            .put("client.transport.sniff", true)
            .put("client.transport.ping_timeout", propertiesUtil.elasticsearchDefaultTimeout)  //20seconds
            .put("client.transport.nodes_sampler_interval", propertiesUtil.elasticsearchDefaultSamplerInterval)    //20seconds
            .build();
    client = new TransportClient(settings);

    String[] searchNodes = propertiesUtil.elasticsearchSearchNodeList.split(",");
    for (String searchNode : searchNodes) {
        client.addTransportAddress(new InetSocketTransportAddress(searchNode, 9300));
    }
}

@PreDestroy
private void destroyElasticSearch() {
    client.close();
}

}

Current caching before, it takes about 2-4 seconds.

I would like to receive a response within one second.
What can I do?

Thanks in advance for your answer.

Some hints:

  • use -Des.index.storage.type=mmapfs

  • you use extensively range and sort. This is awfully slow by design.
    Optimize your discretization of dates. More important, if you can, avoid
    range and sort. If you can not avoid it, increase RAM to the size of
    your total field values so you can completely hold your data to sort and
    filter on in memory. This can be done by more nodes or by more RAM per
    node. You have 49GB index size but only 4 * 4 = 16G assigned to ES heap.

  • use faster CPU for more oomph. I also love Opteron HE but they come at
    a price, they are slower in peak speed than their companions.

Jörg

Am 28.03.13 15:19, schrieb SeongJong:

Current problem, the search speed is slow.
About the system under specification and set lists.

System Specifications
Number of nodes: 4
CPU model: Quad-Core AMD Opteron(tm) Processor 2347 HE (1900 MHz)
CPU total logical cores: 8
Memory : 8G * 4

Elasticsearch Settings (Elasticsearch version is 0.20.6)
index.number_of_shards: 8
index.number_of_replicas: 1
node.master: true
node.data: true
bootstrap.mlockall: true
transport.tcp.compress: true
network.tcp.keep_alive: true
cluster.routing.allocation.node_initial_primaries_recoveries: 1
cluster.routing.allocation.node_concurrent_recoveries: 1
cluster.routing.allocation.cluster_concurrent_rebalance: 1
index.shard.recovery.concurrent_streams: 2
cluster.routing.allocation.disable_allocation: false
cluster.routing.allocation.disable_replica_allocation: false
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
index.cache.field.type: soft
index.term_index_interval: 64
index.refresh_interval: 30

Elasticsearch Start option
bin/elasticsearch -Xmx4G -Xms4G -Des.index.storage.type=memory

Documents Info
size: 49gb (98.2gb)
docs: 18214105 (18214590)

Search Condition

  1. Were routing to each user ID.
  2. Java Code
    FilterBuilder[] reporterFilters = null;
    FilterBuilder filters = null;

SearchRequestBuilder builder = client.prepareSearch(index);
builder.setFrom((Integer) params.get("searchFrom"));
builder.setSize((Integer) params.get("searchCnt"));
builder.setTypes(type);
builder.addSort((String) params.get("sortField"), (Boolean)
params.get("ascending") ? SortOrder.ASC : SortOrder.DESC);

//Set date range filter
if (params.get("fromDate") != null) {
filters =
FilterBuilders.rangeFilter("regDate").from(params.get("fromDate")).to(params.get("toDate"));
}

//Set category filter
if (params.get("categoryId") != null) {
filters =
FilterBuilders.andFilter(FilterBuilders.termFilter("categoryId",
params.get("categoryId")), filters);
}

//Set query filter
if (params.get("query") != null) {
builder.setQuery(QueryBuilders
.queryString((String) params.get("query"))
.field("title")
.field("summary")
.field("content")
.defaultOperator(QueryStringQueryBuilder.Operator.AND)
.analyzer("cjk")
.useDisMax(true)
);
}

builder.setFilter(filters);

SearchResponse response = builder
.execute()
.actionGet();

*Current caching before, it takes about 2-4 seconds.

I would like to receive a response within one second.
What can I do?*

Thanks in advance for your answer.

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/I-want-to-improve-the-search-speed-tp4032562.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I sincerely thank for your advice.

I'll try to test it again after memory expansion.