Bulk file with 5 indexes

i have created a bulk file with 5 indexes and generated load from one server to another server using a script (input of script is bulk.json file) ..
1.why only four indexes are created(verified using elasticsearch-HQ plugin)
2.there is a timeout error after 4000 users
3.also the http request rate per second is decreasing on more indexes its around 100-170 per sec with 5 test cases which is less
4.theres a session failure nearly around 4000 users
5.how can i know the inidexing rate per sec any visuaization plugin?
6.how many users can elasticsearch handle at once?
my script file
#include "ns_string.h"

void flow()
{
ns_web_url ("NF-PT",
"URL=http://localhost:9200/_bulk" ,
"METHOD=PUT" ,
BODY_BEGIN,
"$CAVINCLUDE$=/home/netstorm/work/bulk_sample2.json",
BODY_END

   );

Hi @srikanth_muddu,

1.why only four indexes are created(verified using elasticsearch-HQ plugin)

This is probably related to issue number 2?

2.there is a timeout error after 4000 users

What do you mean by "users"? Do you mean that you apply load from 4000 clients against the cluster or does your bulk data file contain user data and you actually mean that there is a timeout error after you've sent 4000 lines of JSON?

3.also the http request rate per second is decreasing

Not sure what you are experiencing here. Some variance is expected but that's hard to tell based on the information you provide.

5.how can i know the inidexing rate per sec any visuaization plugin?

You cannot. You have to calculate it yourself in your benchmark driver.

6.how many users can elasticsearch handle at once?

This depends on a whole lot of factors: what data do you index, what features you use, how many nodes, your traffic mix, your SLA, hardware configuration, OS configuration, network, etc. etc. etc.

I also want to mention that we have already solved a lot of problems that you encounter with Rally. It should help you focus on specifying and running your benchmark rather than having you rewrite everything from scratch. I really recommend that you check it out (Disclaimer: I am the main author of Rally so I may be biased ;)). If you have any questions about Rally, just post them here in this forum.

Daniel

4000 hits onto a server where each hit has a input(_bulk file) containing around 1500 documents with 5 indexes and auto id generation
my bulk contains sample log data
how many docs can bulk have in it
thanks il go through rally and get back if im stuck ...
please dont mind if these are silly .

Hi @srikanth_muddu,

This depends on the size of individual documents. If you index small documents like log lines or metrics records you can have a larger bulk size than if you try to index larger documents like contents of a book. See How Big is Too Big? and Indexing Performance Tips in the Definitive Guide.

Sure. :slight_smile:

Daniel