Handling elastic indexes correctly, without slowing performance


(dina) #1

I have an elastic instance,

memory size: 2g.

I have 5 apps (lets call them aa, bb, cc, dd, ff) each has production and sandbox environment
each app logs approximately 50,000 messages a day (production environment)
production logs should be kept forever, about sandbox logs I don't really care and they are deleted every week.

currently this is my index:
app_name-environment-YYYY.MM.dd
e.g. : aa-production-2017.04.25

on kibana I created 5 index-patterns:

aa-*
bb-*
cc-*
..

since daily indexing caused high memory pressure & slowness I would like to get some advice on using elastic correctly.

is using monthly indexes recommend? will this slow my search?
or I should use weekly indexes? note I have 5 apps, which I will need their logs for long term

what should I do to make sure I get good performance and make sure I use elastic the best way?


(Christian Dahlqvist) #2

Having daily indices per application and environment will result in a lot of unnecessary overhead, especially for such a small node. For the data you intend to keep indefinitely a longer time period is definitely recommended. A monthly, or maybe even yearly (~20M documents is not a lot for an index), index with a single primary shard should work well. For the sandbox environment it may be suitable to keep daily indices due to the short retention period, but you may be able to keep these longer too if you used a weekly air monthly index here too.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.