Elastic search configuration for extensive search and read/write and indexing operations with heavy number of users

Elastic Stack Elasticsearch
1

Hello,

We need information regarding elastic search configuration for our application environment.

We use elastic search very extensively for our application for search and indexing.

Our application is using Liferay (portal development) with Java as development platform.

We are facing slowness in application related to elastic search when more users are using the application. In application we do a lot of index read and write with several operations.

As per our application monitoring we see many thread are in waiting status for read/write indexing.

Please suggest the best elastic search configuration as per below mentioned application details.

Application: Portal developed on Liferay with Java
Database :- Oracle
Java Version :- Open JDK 8
Search Engine :- Elastic Search
Elastic Search Version :- 6.1.3
Operation System: - Linux
Number of nodes :- 2/4
CPU Cores :- 4 cores
Number of concurrent users at a time :- 100 - 200 (2 nodes) / 250 - 400 (4 nodes)

Elastic Search yml configuration :-

bootstrap.memory_lock: true

discovery.zen.minimum_master_nodes: 1

indices.query.bool.max_clause_count: 10240

thread_pool:
search:
size: 40
queue_size: 500
min_queue_size: 10
max_queue_size: 1000
auto_queue_frame_size: 2000
target_response_time: 2s
index:
queue_size: 10000
bulk:
queue_size: 500

Thanks,
Saurabh

2

How much data do you have in the cluster?

How many indices and shards?

Are you using attached SSD storage?

What does the workload look like? What type of queries are you using? How many queries per second to you need to support?

3

@Christian_Dahlqvist

Hello ,

We have near to 6 GB data in cluster.

Indices settings are default. We have 2 shards.

We use LINUX servers (not laptops) with no additional SSD storage.

Workload mostly includes Searching articles/ documents and media, searching users , updating users (write actions) . Also it includes getting list of articles by type . etc

As per our monitoring I see queries are mostly search ,
org.elasticsearch.action.bulk.BulkRequest (count of 1400 for a 10 minute test),
org.elasticsearch.action.admin.indices.refresh.RefreshRequest (count of 900 for a 10 minute test)
etc.

Also we see LIMIT EXCEEDED BUCKET in monitoring log queries for elastic search (count of 1841 for a 10 minute test).

Thanks,
Saurabh

4

How much RAM does each server have? What is your heap size? Is there anything else running on these hosts?

5

@Christian_Dahlqvist

Hello,

The total RAM size is 32012 MB.

MaxHeap Size for Liferay Tomcat Servers is 8192 MB.

We have another tomcat and Elastic search running on same host.

Thanks,
Saurabh

6

What type of storage does the node have? What does disk I/O and iowait look like?

7

@Christian_Dahlqvist

The Disk Storage type is Local Storage.

Not sure what iowait is.

Also one additional info is the refresh interval of elastic is 60 seconds.

Thanks,
Saurabh

8

@Christian_Dahlqvist ,

Hello

did you get chance to check my updates comment ?

Please let me know if you need additional information .

Thanks,
Saurabh

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.