Need help to convert fielddata to doc_values

[pastorsx (simon)]

Hi all,

I have a cluster and we use ES to index our data. We then use Kibana to display those all in different dashboards, but we think the memory usage by ES is just way too much and we cannot fix the issue.

I read about doc_values vs fielddata... so my question is how can we force data to be sent as doc_values. ES is fed by either filebeat or metricbeat.... we are getting about 10000 hits/min from filebeat and 34000 hits/min from metricbeat.

How can I force data to be doc_values? Where can that be setup?.. I saw templates from logstash to ES that could be used... although it seems per .... but I am not using logstash and it seems to be done by fields but metricbeat has 1673 fields and filebeat 402.... that can't be updated indidually, there must be something big I am missing. Can you please point me out in the right direction?

thank you in advance

[mobidyc]

Hello,

it depends of your *beat version.
this doc should help a lot:
https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html

before version 5/6, the fields were not "text" and "keyword" but "string" (analyzed or not).

[wangqinghuan (Wang)]

I am using Filebeat 6.x and ES 6.x, all fields which support doc values have them enabled by default