What does CPU and disk I/O looking like during indexing?
CPU and IO are around 20-30% . Below is the iostat output
avg-cpu: %user %nice %system %iowait %steal %idle
14.30 0.00 2.03 0.84 0.00 82.83
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 665.00 0.00 291.00 0.00 11808.00 81.15 1.01 3.46 0.00 3.46 1.03 30.00
How large are your documents?
There are only 10 unique documents in whole benchmark. Each document contains maximum of 10 fields with key of 10 characters and value of 50 characters. This means that a document has max of 600 characters. And documents are sent to ES in a bulk of 5000 documents continuously(each batch contains 5000 docs).
What type of data do they contain?
Each character is a alphabet
What do your mappings look like?
It is dynamic mapping. I haven't created any static mappings.
How are you loading the data?
I am using elasticsearch python module Elasticsearch().bulk() to push bulk requests
Here is the output after 30 second run
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "cluster",
"timestamp" : 1498770136805,
"status" : "green",
"indices" : {
"count" : 10,
"shards" : {
"total" : 10,
"primaries" : 10,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"primaries" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 905000,
"deleted" : 0
},
"store" : {
"size_in_bytes" : 100841187,
"throttle_time_in_millis" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"total_count" : 0,
"hit_count" : 0,
"miss_count" : 0,
"cache_size" : 0,
"cache_count" : 0,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 55,
"memory_in_bytes" : 1652155,
"terms_memory_in_bytes" : 1430943,
"stored_fields_memory_in_bytes" : 53352,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 154880,
"points_memory_in_bytes" : 0,
"doc_values_memory_in_bytes" : 12980,
"index_writer_memory_in_bytes" : 0,
"version_map_memory_in_bytes" : 0,
"fixed_bit_set_memory_in_bytes" : 0,
"max_unsafe_auto_id_timestamp" : -1,
"file_sizes" : { }
}
},