Getting timeout exception while inserting data to elastic index

Hello All,

From past 1 week I am getting below exception for elastic index and unable ti find RCA. It showing timeout and not able to find what could be reason for timeout while inserting data to index

Even in elastic logs no traces found for excetion. Basically an api is hit to generate data.

<$0d272797> org-springframework.data.elasticsearch.ElasticsearchException: Error while index for request: index ((mic_sfde_cache] (sfdecontext) (c0b1b2a0-8553-4b4a-a763-fbbd6ebBb221], source l ("id"="c0b1b2a0-8553-4b4a-a763-fbbd6ebBb221", "serviceName": "RCU_KARZA_DISCOVER", "sfdcEnv": "uat", "chidrPartyAckId"="c0b1b2a0-8553-4b4a-a763-fbbd6ebBb221", "sfdcId"="a2DC400000BzG

hSMAV", "requestPayLoad" :null, "sourceId": "00DC4000000bCevMAE"||} at org.springframework.data.elasticsearch.COre.ElasticsearchRestTemplate.index(ElasticsearchRestTemplate.java:699) at

org.springframework.data.elasticsearch.repository-support.AbstractElasticsearchRepository-save (AbstractElasticsearchRepository-java:176) at

Java.1ang.reflect. Method.invoke(Method.java:498) at org.9pzingframework.data.repository.coze.gupport.RepositoryCompositionsRepositoryFragments.invoke(RepositoryComposition.java:359)at or9-springframewozk.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition-java:200)at

org.springframework.data.repository.core.suPPort.RepositoryFactorySupport$ImplementationethodBxecutionIntezceptoz.invoke (RepositoryFactorySupport.java:644)atorg.springfromewozk.aop-fzamework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at

org-3pringframework.data.repository.core.9uPPort.RepositoryFactorySupportSQueryExecutozMethodInterceptoz.doInvoke(RepositoryFactorySupport.java:608)ator9-3pzingframewozk.data.repository.core.support.RepositoryFactorySupportsQueryExecutozMethodInterceptoz.1ambdasinvoke$3 (RepositoryFactorySupport.java:595) atorg-apringtramewozk.data.repository.core.support.RapositoryFactorySupport$QueryExecutozMethodInterceptor.invoke (RepositoryFactorySupport.java:595)atorg.gpringframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:186) atorg.springframewozk.data.projection.DefaultMethodInvokingMethodInterceptor.invoke (DefaultMethodInvokingMethodInterceptoz.java: 59) atorg.springframewozk.aop.framewozk.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:186) atorg-springfzamework.aop.interceptor.BxposeInvocationInterceptoz.invoke (ExposeInvocationInterceptor.java:93) atorg .springtramework.aop.tramework.RetlectiveMethodinvocation.proceed (ReflectiveMethodInvocation. java:186) at

org.springframework.data.repository.core.gupport.SurroundingTransactionDetectozMethodInterceptoz.invoke(SurzoundingTransactionDetectozMethodInterceptor.java:61)atorg-springframewozk.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:186) at

org-springframework. aop.framerork.JdkDynamd cAopProxy.invoke (JdkDynami chopProxy.java:212) at com.sun.proxy. $Proxy181.save (Unknown Source) at

com.yb1.genie.bureau.sezvice.impl.CustomerVezificationServiceImpl.handleResponsevithstatus (CustomexVerificationServiceImp1.java:102)atcom.yb1.genie.bureau.service.imp1.CustomezVezificationServiceImpl.vezifyCustomer(CustomerVezificationServiceImp1.java:74) at

org.springframework.aop.framewozk.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint (CglibAopProxy.java:750) atorg.springframework.aop.framework.ReflectiveMethodInvocation.proceed (ReflectiveMethodInvocation.java:163) at|

org.springfzamewozk.aop.framewozk.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ator9.springtramewozk.aop.framewozk.CglibAopProxy$DynamichdvisedInterceptor.intercept (CglibAopProxy.java:689) atcom.yb1.genie.Dureau.service.impl.CustomerVeriticationServiceimpL$$BnhancexBySpringvGbuB$$:3901938.verityCustomer () acom.yb1.genie.bureau.contzo1ler.CustomerVezificationContro1lex.1ambdafverifyCustomer$D (CustomerVerificationContro1lez.java:67)at

jeva.utii.concurrent.CompletableFuture$hsyncRun.run (CompLetablenuture.java:1626) at ox9.springtramework.cloud.sleuth.instrument.async.TzaceRunnable.run(TraceRunnable.java:67)etjeva.ut11.concurrent.ThreadBoo1Bxecutor.runNorkex(ThreadBoo1Executor.java:1149) at java.uti1.concurrent.ThreadPooLBxecutoz$Wozkez.zun (ThreadPoo1Bxecutor.java:624) atjava.1ang.Thread.zun (Thread.jave:74B)Caused by: <#3fe709e1> java.1o.IOBxception:1istener timeout aftex waiting for (30000] ma ator9.elasticseazch.client.RestClientsSyncResponselistener .get (RestClient.jeva:899) at org.elasticsearch.client.RestClient performRequest (RestClient.java:227) atorG.elesticseazch.client.RestHighleve1Client.pezformRequest(RestHighleve1C1ient.java:1256) atorg.elasticsearch.client.RestHighleve1Client.pezfozmRequestAndParseintity (RestHighieve1Client.java:1231) at

org.springtremework.data.elesticsearch.core.BlasticsearchRestTemp1ate.index (E1asticsearchRestTemplate.java: 697).. 38 common trames omitted

sun.reflect.GeneratedMethodAccessOr256.invoke (Unknown Source) at sun.reflect.DelegatingMethodAccessorImp1.invoke (DelegatingMethodAccessorImpl.java:43) at

com.ybl.genie.bureau.service.imp/.CustomerVerificationServiceImpl.handleSuccegs (CustomerVerificationServiceImpl.java:147) at

com. ybl.genie.bureau.service.imp1.CustomerVerificationServiceImp1$$FastClassBySpringCGLIB$$9ec37061. invoke () at

ora.springframework.cglib.proxy.MethodProxy-invoke (MethodProxy-java:218) at

org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at

org.elasticsearch.client.RestHighLevelClient. index (RestHighLevelC1ient.java: 587) at

Thanx

Hello,

Could you please try to share more information?
your normal indexing is working fine and only it is failing via API & giving timeout?
you have checked the cluster is up & running fine?
I see you have checked elasticsearch logs & there are no error. Could you please share how you are calling the API , what is that API , more details in order to understand what is the issue?
Thanks!!

So we’re running hadoop jobs throughout the day and it’s noticed that between 2:30 to 4 when these jobs are run then is high indexing rate through stack monitoring and some documents gets rejected with above exception.
Unable to find root cause how could I avoid happening this thing.
Cluster state is green and all nodes are up.