Sorry Dadoonet! I provided full source code below which related to ElasticSearch and also i defined scroll object which i did not provided in earlier code.
Issue here is: if Spring boot API is not received any request(keep idle) for more than 30 mins to 1 hr, then for first few request alone getting "java.io.IOException: listener timeout after waiting for [30000] ms" exception then its working and retriveing data from backend ES.
ElasticSearch Config class:-
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.thehartford.rdm.RDMConstants;
@Configuration
public class ElasticsearchConfig {
@Bean(destroyMethod = "close")
public RestHighLevelClient client() {
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials("John","JohnES@123"));
RestClientBuilder builder = RestClient.builder(new HttpHost("a4423e1npvml001", 54321, "https"))
.setRequestConfigCallback(reqConfigCallBack -> reqConfigCallBack .setConnectTimeout(30000).setSocketTimeout(30000)) .setMaxRetryTimeoutMillis(30000)
.setHttpClientConfigCallback(httpClientConfigCallback -> httpClientConfigCallback .setDefaultCredentialsProvider(credentialsProvider));
return new RestHighLevelClient(builder);
}
}
ElasticSearch Util class method which directly invoked from Spring boot controller
import org.apache.log4j.Logger;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchScrollRequest;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.Scroll;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
@Component
public class ElasticSearchUtil {
private static final Logger LOGGER = Logger.getLogger(ElasticSearchUtil.class);
final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
@Autowired
private RestHighLevelClient client;
private String excludeDefaultFields = "@timestamp,agent,ecs,file,host,input,log,message";
private String excludeRDMFields = "VersionID,-,ReviewDate,ExpiryDate,StandardID";
private int pagecount =100;
private List<String> tempList = Arrays.asList("action,resource,systemShortName,contextName".split(","));
public List<Object> searchWithES(String indexName, Map<String, String> requestParams) throws IOException {
SearchRequest searchRequest = new SearchRequest(indexName.toLowerCase());
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
BoolQueryBuilder subBool = QueryBuilders.boolQuery();
// Search with enterprise set/Index name which should not have "_" or
// Search with source set to validate MISSING field name
if (requestParams != null && requestParams.size() > 0) {
requestParams.forEach((k, v) -> {
if (!tempList.contains(k)) {
if (indexName.endsWith("_map")) {
subBool.must(
QueryBuilders.matchQuery("json_from_csv.Source_" + k + ".keyword", v == null ? "" : v));
} else {
subBool.should(QueryBuilders.matchQuery("json_from_csv." + k + ".keyword", v == null ? "" : v));
}
}
});
searchSourceBuilder.query(subBool);
} else if (!indexName.contains("_") || requestParams == null) {
searchSourceBuilder.query(QueryBuilders.matchAllQuery());
}
searchSourceBuilder.fetchSource(null, excludeDefaultFields.split(","));
searchSourceBuilder.size(indexName.contains("_") && requestParams == null ? 1 : pagecount);
searchRequest.scroll(scroll);
searchRequest.source(searchSourceBuilder);
List<Object> outputList = new ArrayList<>();
LOGGER.info("Search JSON query: {}\n" + searchRequest.source().toString());
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();
while (searchHits != null && searchHits.length > 0) {
Arrays.stream(searchHits).forEach(hit -> outputList.add(hit.getSourceAsMap().get("json_from_csv")));
if (searchSourceBuilder.size() == 1) {
break;
}
SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
scrollRequest.scroll(scroll);
searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
scrollId = searchResponse.getScrollId();
searchHits = searchResponse.getHits().getHits();
}
return outputList;
}
Exception trace:-
15:06:32,787 ERROR [com.thehartford.rdm.util.ElasticSearchUtil] (default task-765) Exception when fetching from ElasticSearch::java.io.IOException: listener timeout after waiting for [30000] ms
15:06:32,877 WARN [org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver] (default task-765) Failure in @ExceptionHandler public org.springframework.http.ResponseEntity<com.thehartford.rdm.dto.RDMServiceResponse> com.thehartford.rdm.controllers.CustomRestExceptionHandler.handleMissingParams(javax.servlet.http.HttpServletRequest,com.thehartford.rdm.exception.RDMException): java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:162)
at io.undertow.conduits.BytesSentStreamSinkConduit.write(BytesSentStreamSinkConduit.java:76)
at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:259)
...