Long time on loaded plugins from java client

[nugusbayevkk (nugusbayevk)]

Hi guys, i have problem or is it normal?

in this example after run i see that a lot of time spended on LOADED PLUGINS. Is it correct?
It took about 600msec.

311
318
no modules loaded
loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
loaded plugin [org.elasticsearch.transport.Netty3Plugin]
loaded plugin [org.elasticsearch.transport.Netty4Plugin]
after add first node935
after add second node947
after query978
after getHits982
kz.test.HandledEvent@60baef24
kz.test.HandledEvent@61533ae
kz.test.HandledEvent@732bb66d
kz.test.HandledEvent@720653c2
kz.test.HandledEvent@45f24169
kz.test.HandledEvent@6ad5923a

difference between 318 msec and after add first node935
about 600 msec.

Can you give me any advice how to avoid it?
in earlier version i didn't notice this behavior.

This is my configuration:
public class Test {

public static void main(String[] args) throws UnknownHostException {
    long millis = System.currentTimeMillis() % 1000;
    System.out.println(millis);
    Settings settings = Settings.builder().put("cluster.name", "eventhandler-main-db")
            .build();
    long millis2 = System.currentTimeMillis() % 1000;
    System.out.println(millis2);
    TransportClient client = new PreBuiltTransportClient(settings);
    client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("grafana-dashboard-d"), 9300));
    long millis3 = System.currentTimeMillis() % 1000;
    System.out.println("after add first node"+millis3);
    client.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("grafana-dashboard-d"), 9301));
    long millis4 = System.currentTimeMillis() % 1000;
    System.out.println("after add second node"+millis4);
    List<HandledEvent> events = new ArrayList<HandledEvent>();
    ELKQueryBuilder elkqb = new ELKQueryBuilder();
    QueryBuilder qb = elkqb.createQuery("_all", "2016-12-05T16:42:00+06:00", "2016-12-05T17:42:00+06:00", "_all");
    
    SearchResponse scrollResp = client.prepareSearch("events-2016.12.05").setTypes("docs")
            .setSize(10000)
            .setTimeout(new TimeValue(1000))
            .setQuery(qb)
            .addAggregation(
                    terms("bySource").field("source").size(50)
                    .subAggregation(terms("byTheme").field("theme").size(10000)
                            .subAggregation(sum("eventsCount").field("cnt"))
                            .subAggregation(sum("readEventsCount").field("isRead"))
                            .subAggregation(max("latest_event").field("created_time"))
                            .subAggregation(min("first_event").field("created_time"))
                    ))
            .get();
    long millis5 = System.currentTimeMillis() % 1000;
    System.out.println("after query"+millis5);
    Terms sourcce = scrollResp.getAggregations().get("bySource");
    Collection<Terms.Bucket> sourceBuckets = sourcce.getBuckets();
    for (Terms.Bucket sourceBucket : sourceBuckets) {
        Terms themme = sourceBucket.getAggregations().get("byTheme");
        for (Terms.Bucket themeBucket : themme.getBuckets()) {
            HandledEvent event = new HandledEvent(sourceBucket.getKeyAsString(), themeBucket.getKeyAsString(), (double) themeBucket.getAggregations().get("eventsCount").getProperty("value"),
                    (double) themeBucket.getAggregations().get("readEventsCount").getProperty("value"), (double) themeBucket.getAggregations().get("latest_event").getProperty("value"),
                    (double) themeBucket.getAggregations().get("first_event").getProperty("value"));
            events.add(new HandledEvent("test", "test", 3, 3, 3, 3));
            events.add(event);
            event = null;
            themeBucket = null;
        }
        themme = null;
        sourceBucket = null;
    }
    long millis6 = System.currentTimeMillis() % 1000;
    System.out.println("after getHits"+millis6);
    events.forEach(System.out::println);
}

}

[nugusbayevkk (nugusbayevk)]

in this example i was using elasticsearch version 5.0.2.
earlier i used 2.3.4.

[spinscale (Alexander Reelsen)]

Hey,

yes, the TransportClient is doing some more set up work and thus needs a bit. You should create a transport only once in your application as it is thread-safe and reuse it - so there is no need to worry about start up times.

--Alex

[nugusbayevkk (nugusbayevk)]

Alex, and last question:
how do you think: is it correct if i made by like this:
I create commentClient object, and then will use him in any methods of my class?

I tried like this, and took a great results, the time of queries execution decreased in 2-3time. earlier in average work 70-100ms, now 25-40 ms.

TransportClient commonClient = new DBConnector().getElasticSearchConn();

@Override
public void createEvent(Event event) {
 
    String today = df.format(new Date()).toString();
    try {
        event.setCnt(1);
        event.setIsRead(0);
        DateTime datetime = new DateTime();
        IndexResponse responseUser = commonClient.prepareIndex("events-" + today, "docs")
                .setSource(jsonBuilder()
                        .startObject()
                        .field("created_time", datetime.toString())
                        .field("cnt", event.getCnt())
                        .field("isRead", event.getIsRead())
                        .field("source", event.getSource())
                        .field("theme", event.getTheme())
                        .field("description", event.getDescription())
                        .endObject()
                ).get();
    } catch (IOException ex) {
        Logger.getLogger(EventServiceImpl.class.getName()).log(Level.SEVERE, null, ex);
    } finally {
      
    }
}

@Override
public List<Event> getEvents(String source, String period_begin, String period_end, String eventType) {
  
    List<Event> events = new ArrayList<Event>();
    ichecker = new IndexChecker();
    try {

       
        QueryBuilder qb = elkqb.createQuery(source, period_begin, period_end, eventType);

        SearchResponse scrollResp = commonClient.prepareSearch(ichecker.getExistedLastIndices(commonClient)).setTypes("docs")
                .addSort(fieldSort("created_time").order(DESC))
                //                    .setScroll(period) //new TimeValue(60000)
                .setTimeout(new TimeValue(1000))
                .setQuery(qb)
                .setSize(10000).execute().actionGet();
        for (SearchHit hit : scrollResp.getHits()) {
            Event event = new Event(hit.getSource());
            events.add(event);
            hit = null;
        }
        qb = null;
        scrollResp = null;
        ichecker = null;
    } catch (SearchParseException ex) {
        Logger.getLogger(EventServiceImpl.class.getName()).log(Level.SEVERE, null, ex);
    }  catch (ParseException ex) {
        Logger.getLogger(EventServiceImpl.class.getName()).log(Level.SEVERE, null, ex);
    } finally {
       
    }
    return events;
}