ELK for 30k eps - memory problem

Hi em01,

By default, your version of elasticsearch is building fielddata every time you access the timestamp field for aggregation or sorting, which happens pretty much every time you open any kibana dashboard or execute a search query that sorts by timestamp. Because building fielddata is an expensive process, elasticsearch caches it in memory. Please see https://www.elastic.co/guide/en/elasticsearch/guide/master/fielddata.html and https://www.elastic.co/guide/en/elasticsearch/guide/master/doc-values.html for more informaiton.

So, to avoid this problem, you can either give more memory to your elasticsearch cluster by adding more nodes or you can switch to doc values for the timestamp field. See Using Doc Values for some pointers on how to enable doc values on the logstash level.