Hi, I wrote a program for deleting records according to their timestamp from an elasticsearch node running on an EC2 machine. When I run the program from my local machine, I can connect to the ES node via public IP address and it works as expected. If I try to execute the same program from another EC2 instance within the same availability zone and security group I get an empty result of the execution.
This is how i connect to ES:
TransportClient transportClient = new TransportClient();
this.esClient = transportClient.addTransportAddress(new InetSocketTransportAddress(es_address, 9300));
This is the relevant part of code
FilterBuilder fb = new IndicesFilterBuilder(new RangeFilterBuilder("timestamp")
.from(params.get("start_date"))
.to(params.get("end_date"))
.includeLower(false)
.includeUpper(true), "test").noMatchFilter("none");
//search
SearchResponse sr = this.esClient.prepareSearch()
.setQuery(new MatchAllQueryBuilder())
.setPostFilter(fb)
.setScroll(new TimeValue(10000))
.setSize(10000)//keep the scroll context alive for 10 seconds
.get();
LOG.info("Search request completed in : " + sr.getTook() + " "+ sr.getHits().getTotalHits());
//scroll the result
boolean scroll = true;
while(scroll)
{
for(SearchHit hit : sr.getHits().getHits())
{
bp.add(new DeleteRequest(hit.getIndex(), hit.getType(), hit.getId()));
}
sr = this.esClient.prepareSearchScroll(sr.getScrollId())
.setScroll(new TimeValue(10000)) //keep the scroll context alive for 10 seconds
.get();
LOG.info("Search request completed in : " + sr.getTook());
if (sr.getHits().getHits().length == 0) { //no more results
scroll = false;
}
}
According to the program output, I can connect to the node, but from my local machine the search request returns hits and then deletes them, instread from another EC2 instance the same search returns 0 documents. I am not using AWS plugin.
PS. I have a spark and a storm cluster running in the same VPC feeding correctly the same node.