Splitting the elastic index to am and pm

i have an elastic index on date basis and it should get 1B documents/day. in the current scenario i am able to get only 500-600M and it is too slow [ like getting yesterday's logs today to elastic] . I am thinking of splitting the index into AM and PMso as to improve performance. but i am not too sure if elastic can differentiate the difference between am and pm. I have a shell script that creates the mapping a day before, date=date --date="+1 day" +%Y-%m-%d
can i just append AM/PM to date=date --date="+1 day" +%Y-%m-%d-%p .

What is too slow? Indexing? Querying?

Indexing as well as querying. But at this point i am only worried about indexing.i have a 5 node cluster with c4.8*large cluster instance type and have 2 indexes per day with one of them getting 100M/day and the other 1billion/day. no issues with the one getting 100M/day .

How many shards have you got configured for the larger index? How large do the shards get?

I have 5 shards for the larger index. each shard is showing around 150gb for yesterday.

You might benefit from using the rollover index API as this allows you to set a target size with respect to number of documents and/or time. You could then rollover to a new index based on the shard size rather than strictly by time, which would make it easier to deal with increasing or peaky loads.

Thanks Christian.Will implement this and post the results.

followed the steps as in the link you provided me....
from the shell script did the following

curl -X PUT 'http://server:9200/test-'$date-1 -d '{
"aliases": {
"test": {}

checked from the command line curl -XGET http://server:9200/test/_alias

inserted data into the index using curl -XPUT http://server:9200/test/log/1 -d '{"message" : "Test"}' - 1 document is inserted into the index

tried to rollover

curl -X POST 'http://server:9200/test/_rollover' -d '{
"conditions": {
"max_docs": 1

when i run the above i am getting {"error":"InvalidTypeNameException[mapping type name [rollover] can't start with '']","status":400}. Can you let me know how i can resolve this?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.