Search Request are Getting Socket Timeout

Hi Team,

We have an index containing 365 days data.

We have started reprocessing existing 1-Day data (i.e reprocessing includes-> Insert New Data + then delete the Old data), we observed that while reprocessing is going on Search Queries are getting Socket Timeout because as reprocessing is going on Search Queries are taking more time than usual.

Ex:
Existing Data counts = 3370109 (for 1-Day)
New Data = 3370109 (New Data for 1-Day)

Reprocessing includes -> Insert New Data (Its Spark Write) + then Delete Old Data (Its HighLevelRestClient Delete By Query)

So while reprocessing is going on Search Queries are getting timeout and we should not increase the SocketTimeOut because its an user interface where should not see any latency in seeing the data.

I am aware that Elastic Search allows for Data Reads while data process is going on, Could you please help us how to Read the data from elastic search cluster without Socket Timeout or without any latency for Search Requests (data read) while reprocessing is going on.

Thank in advance.

Have you looked at identifying which resource is causing the timeouts? I would recommend first looking to see if disk I/O might be the limiting factor as it often is. Also look at CPU and for evidence of long or frequent GC in the logs.

Here the problem is We are reading the Same reprocessing data while reprocess is going on. i.e data reads are taking time while our data reprocessing is going on which leads to the SocketTimeOut.

Ex:
Existing Data counts = 3370109 (for 1-Day)
New Data = 3370109 (New Data for 1-Day)

Can we increase I/O factor? if yes could you please help me how increase it? And is it going to help us in reading the data faster while reprocessing is going on?

First step is to identify what is limiting performance and causing latency. What type of storage do you have, e.g. local SSD? Have you looked at disk I/O, e.g. using iostat, while you are performing this operation?

Hey Christian,

We are using AWS Instance Types: c5.4xlarge.elasticsearch
Data nodes storage type: EBS
EBS volume type: Provisioned IOPS (SSD)
EBS volume size: 500 GiB
Provisioned IOPS: 5000
And
"index.store.type": "niofs"

Do you have graphs showing disk I/O over time that you can correlate with the times the jobs run? Why have you got the niofs setting in place?

Hey Christian,

I don't have graphs, if you need i can run one more time & i can send you.

And i have gone through the ES Docx and configured the 'niofs'

The NIO FS type stores the shard index on the file system (maps to Lucene NIOFSDirectory ) using NIO. It allows multiple threads to read from the same file concurrently.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.