We are currently evaluating Elasticsearch as our solution for Analytics.
The main driver is the fact that once the data is populated into
Elasticsearch, the reporting comes for free with Kibana.
Before adopting it, I am tasked to do a performance analysis of the tool.
The main requirement is supporting a PUT rate of 500 evt/sec.
I am currently starting with a small setup as follows just to get a sense
of the API before I upload that to a more serious lab.
My Strategy is basically, going over CSVs of analytics that correspond to
the format I need and putting them into elasticsearch. I am not using the
bulk API because in reality the events will not arrive in a bulk fashion.
Following is the main code that does this:
// Created once, used for creating a JSON from a bean
ObjectMapper mapper = new ObjectMapper();
// Creating a measurement for checking the count of sent events vs
// ES stored events
AnalyticsMetrics metrics = new AnalyticsMetrics();
metrics.startRecording();
File dir = new File(mFolder);
for (File file : dir.listFiles()) {
CSVReader reader = new CSVReader(new FileReader(file.getAbsolutePath()), '|');
String [] nextLine;
while ((nextLine = reader.readNext()) != null) {
AnalyticRecord record = new AnalyticRecord();
record.serializeLine(nextLine);
// Generate json
String json = mapper.writeValueAsString(record);
IndexResponse response = mClient.getClient().prepareIndex("sdk_sync_log", "sdk_sync")
.setSource(json)
.execute()
.actionGet();
// Recording Metrics
metrics.sent();
}
}
metrics.stopRecording();
return metrics;
I have the following questions:
- How do I know through the API when all the requests are completed and
the data is saved into Elasticsearch? I could query Elasticsearch for the
objects counts in my particular index but doing that would be a new
performance factor by itself, hence I am eliminating this option. - Is the above the fastest way to insert object to Elasticsearch or are
there other optimizations I could do. Keep in mind the bulk API is not an
option for now.
Thx in advance.
P.S: the Elasticsearch version I am using on both client and server is
1.0.0.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2afba995-4ae2-41a5-b395-7a90ea2fcc86%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
