The size of documents and the number and types of fields will affect indexing rate as it will define the amount of work Elasticsearch need to do for each document. The reason for the low CPU usage is however probably that Rally has the ability to partition work and use multiple worker processes bulk indexing in parallell, whereas your script appear to be single threaded. Do you see better resource utilization if you increase the number of your scripts that you run concurrently?
The problem is that when I run eight scripts ,the cpu usage still at 300% - 400%, and the exception [EsRejectedExecutionException[rejected execution (queue capacity 50) ] catched.
Don't jump directly to 8 scripts. Instead start with 2 and slowly increase. You may also want to try with a smaller bulk size. Have you run Rally with this type of events into an index with the same number of shards?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.