Splitting the elastic index to am and pm

Elastic Stack Elasticsearch
1

i have an elastic index on date basis and it should get 1B documents/day. in the current scenario i am able to get only 500-600M and it is too slow [ like getting yesterday's logs today to elastic] . I am thinking of splitting the index into AM and PMso as to improve performance. but i am not too sure if elastic can differentiate the difference between am and pm. I have a shell script that creates the mapping a day before, date=date --date="+1 day" +%Y-%m-%d
can i just append AM/PM to date=date --date="+1 day" +%Y-%m-%d-%p .

2

What is too slow? Indexing? Querying?

3

Indexing as well as querying. But at this point i am only worried about indexing.i have a 5 node cluster with c4.8*large cluster instance type and have 2 indexes per day with one of them getting 100M/day and the other 1billion/day. no issues with the one getting 100M/day .

4

How many shards have you got configured for the larger index? How large do the shards get?

5

I have 5 shards for the larger index. each shard is showing around 150gb for yesterday.

6

You might benefit from using the rollover index API as this allows you to set a target size with respect to number of documents and/or time. You could then rollover to a new index based on the shard size rather than strictly by time, which would make it easier to deal with increasing or peaky loads.

7

Thanks Christian.Will implement this and post the results.

8

followed the steps as in the link you provided me....
from the shell script did the following

curl -X PUT 'http://server:9200/test-'$date-1 -d '{
"aliases": {
"test": {}
}
}'

checked from the command line curl -XGET http://server:9200/test/_alias
{"test-2017-07-11-1":{"aliases":{"test":{}}}}

inserted data into the index using curl -XPUT http://server:9200/test/log/1 -d '{"message" : "Test"}' - 1 document is inserted into the index

tried to rollover

curl -X POST 'http://server:9200/test/_rollover' -d '{
"conditions": {
"max_docs": 1
}
}'

when i run the above i am getting {"error":"InvalidTypeNameException[mapping type name [rollover] can't start with '']","status":400}. Can you let me know how i can resolve this?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.